Hai Dam

We’ve been running  Kubernetes  for deep learning research for over two years. While our largest-scale workloads manage bare cloud VMs directly, Kubernetes provides a fast iteration cycle, reasonable scalability, and a lack of boilerplate which makes it ideal for most of our experiments. We now operate several Kubernetes clusters (some in the cloud and some on physical hardware), the largest of which we’ve pushed to over 2,500 nodes. This cluster runs in Azure on a combination of D15v2 and NC24 VMs. On the path to this scale, many system components caused breakages, including etcd, the Kube masters, Docker image pulls, network, KubeDNS, and even our machines’ ARP caches. We felt it’d be helpful to share the specific issues we ran into, and how we solved them. etcd After passing 500 nodes in our cluster, our researchers started reporting regular timeouts from the  kubectl  command line tool. We tried adding more Kube masters (VMs running  kube-apiserv...

When designing a durable messaging system I took a hard look at using Amazon’s Kinesis as the message storage and delivery mechanism. At first glance, Kinesis has a feature set that looks like it can solve any problem: it can store terabytes of data, it can replay old messages, and it can support multiple message consumers. But if you dig a little deeper you will find that Kinesis is well suited for a very particular use case, and if your application doesn’t fit this use case, Kinesis may be a lot more trouble than it’s worth. In this article, I compare Kinesis with Amazon’s Simple Queue Service (SQS), showing the benefits and drawbacks of each system, and highlighting the difference between data streams and queueing. This article should make clear why we built our durable messaging system using SQS, and why your application might benefit from SQS too. Data Streams - The Kinesis Sweet Spot Kinesis’ primary use case is collecting, storing and processing real-time continuous data ...

Amazon has generated a lot of excitement around their recent release of Athena, an ANSI-standard query tool that works with data stored in Amazon S3. Athena and S3 can deliver results quickly and with the power of sophisticated data warehousing systems. This article covers 9 things you should know about Athena when considering it as a query service. 1. Schema and Table Definitions To be able to query data with Athena, you will need to make sure you have data residing on S3. With data on S3 you will need to create a database and tables. When creating schemas for the data on S3 the positional order is important. For example, if you have a source file with “ID”, ”DATE”, ”CAMPAIGNID”, ”RESPONSE”, ”ROI” and ”OFFERID” columns, then your schema should reflect that structure. “ID”,”DATE”,”CAMPAIGNID”,”RESPONSE”,”ROI”,”OFFERID” “0122”,”December 10, 2015",”123432100A”,”8.43",”A102B”,”4985" “0123”,”December 10, 2015",”543421111A”,”2.61",”A904F”,”3212...

One of the things I love the most with AWS is  auto-scaling . You choose an AMI, set some parameters and AWS will spin instances up and down whenever a threshold is breached. But with all these instances spinning up and down there are some unknowns. For example, what is the IP address of the new instance? Its host name? This can be critical when other components of your infrastructure are dependent on knowing these parameters. I had this problem when I started to use  Zabbix  as the monitoring system. At first it seemed like a complicated one, but Zabbix has a wonderful feature called  Auto Registration  which can be used exactly for this situation. I will try to show how to configure auto registration both on the client (EC2 instance running Ubuntu 14.04) and on the Zabbix server (Zabbix Server 2.4.2). Zabbix-agent Installation and Configuration Let’s start with installing zabbix-agent on the Ubuntu client: 1 2 $ sudo apt-get update $ sud...