Archives: cloud technology

bigdata OLTP , OLAP No ratings yet.

| By | Post comment

row-based vs col based db or format row based –> good for OLTP ( transcation),   e.g: cassendra col based –> good for OLAP (? easy to aggreation etc?), druid Parquet ( column based data format): https://www.jumpingrivers.com/blog/parquet-file-format-big-data-r/ https://www.upsolver.com/blog/apache-parquet-why-use   hadoop: big data storage, what is the alternatives? S3 on cloud? https://www.alluxio.io/learn/hdfs/basic-file-operations-commands/ https://stackoverflow.com/questions/31011078/data-retention-in-hadoop-hdfs   pinot vs cassandra • Read More »

Kubernetes Quick Start No ratings yet.

| By | Post comment

Learning curve for Kubernetes (K8s) could be steep. For most users who are already familiar with basic networking, OS/Linux,  docker container technologies etc, the difficulties , I personally think,  is most likely due to its unique concepts, sometime confusing terms. Based on my learning experience, I wrote a book: Kerbernetes Quick start, hopefully can help • Read More »

openstack 5/5 (1)

| By | Post comment

I spent several days to install openstack on debian 8 at one testing box with two NICs. Finally got a basic working system! It has with a controller node running neutron, glance and horizon, a compute node using KVM. To get much speed, one KVM VM is created as a controller node on that box, at the same • Read More »

docker No ratings yet.

| By | Post comment

Finally I got some chances to package one application using docker, here are some useful commands docker images: docker build –rm -t my_tag docker_dir ( with Dockerfile) docker images , docker rmi ( remove docker image) docker containers docker run -d –publish 9080:80 –publish 1022:22 –name container_name –restart always –voloume host_dir:container_dir -e HOST_UID=$(id -u) $tag • Read More »

Apache Kafka big picture and quick start No ratings yet.

| By | Post comment

What is Apache Kafka? ( big picture)  I found the article http://www.confluent.io/blog/stream-data-platform-1/ ( from Jay Kreps) presented a very good big picture on what Kafka suppose to do: you can use Kafka to build a stream data platform. Here the pictures from that article. The big idea is simple: many business processes can be modeled • Read More »

Ansible quick howto No ratings yet.

| By | Post comment

Ansible is mainly using ssh to manage nodes, it is an agentless architecture different from chef, puppet. here is quick howto to get started: git clone git://github.com/ansible/ansible.git –recursive $ cd ./ansible $source ./hacking/env-setup sudo pip install paramiko PyYAML Jinja2 httplib2 six sshpass $ echo “127.0.0.1” > ~/ansible_hosts $ export ANSIBLE_INVENTORY=~/ansible_hosts simple run: ansible all -m • Read More »

lxc /dev increased from 100k to 500k and some other minor issues No ratings yet.

| By | Post comment

While using lxc container, one annoying thing is the disk size of /dev was limited to 100k by default, with some hardware ( with lxc.autodev=1), the /dev could go beyond 100k. discussed on the github: https://github.com/lxc/lxc/issues/781, followed the suggestion from Stéphane Graber (stgraber) I submitted the patch to increase the /dev is to 500k ( though • Read More »

what is application container, OS container , VM No ratings yet.

| By | Post comment

Application container: docker OS containers: LXC, OpenVZ, Linux VServer, BSD Jails, Solaris Zones VM: KVM, VMware, VirtualBox Good images from https://blog.risingstack.com/operating-system-containers-vs-application-containers/ References https://blog.risingstack.com/operating-system-containers-vs-application-containers/ http://www.itworld.com/article/2915530/virtualization/containers-vs-virtual-machines-how-to-tell-which-is-the-right-choice-for-your-enterprise.html https://www.flockport.com/supercharge-lxc-with-btrfs/   Please rate this rating

AWS Lambda No ratings yet.

| By | Post comment

A working source code of a simple demo of AWS Lambda for Android Phone could be found at: https://github.com/mingewang/AndroidEventGenerator Notes: (1) The above code runs good on the real android phone. (2) Running on the Android Emulator, I always got the error: final LambdaFunctionsInterface myInterface = factory.build(LambdaFunctionsInterface.class); Method threw ‘java.lang.UnsupportedOperationException’ exception. Cannot evaluate $Proxy0.toString() then it • Read More »