Three weeks in the life of Google Cloud Platform (GCP)

I was off on vacation with family for three weeks in August and came back to work to an impressive list of GCP (Google Cloud Platform) news and I thought I’d compile them here:

Product Launches & Announcements

Lots of GA announcements (Generally Available, fully-supported):

• Cloud Shell is now GA, and still free. \o/

• Cloud Datastore v1 API now GA. Full independence from App Engine if you want it! This release includes console enhancement, best practices, storage size calculation, clearer limits, and multi-tenancy guidance via namespaces. Also, it is now serving 15 trillion queries per month. Snapchat, Workiva and Khan Academy are among customers that can be named at this point.

• Google Cloud Bigtable is now GA. This is the fully-managed database service built on Google’s internal Bigtable service with Apache HBase client compatibility. It powers Google Search, Analytics, Maps, Gmail and more. Dataflow, Dataproc, Pub/Sub and BigQuery are all well integrated with Bigtable and Spotify, FIS and Energyworx are all using it already.

• Cloud SQL Second Generation is now GA. It offers MySQL 5.7, point-in-time-recovery (PITR), automatic storage re-sizing and failover replicas setup with a single click. It’s 7x faster and has 20x more storage capacity than its predecessor. See also the post for competitive benchmarks. Here are short videos on how to use Cloud SQL with App Engine and with Compute engine.

• Customer-Supplied Encryption Keys for Compute Engine goes GA. Get both control over data AND agility of the cloud. Now available in 14 countries.

The Cloud team is also exploring some (relatively) new and important territory :

• “Making ASP.NET apps first-class citizens on Google Cloud Platform”. Showing Microsoft developers lots of love with Windows Server on GCE, ASP.NET image, SQL Server on GCE, Google services via NuGet, Cloud Tools for Visual Studio, Cloud Tools for PowerShell and ASP.NET Core support coming soon. Check out this short one-minute intro video.

Some important product enhancements…

… in Compute Engine :

• Preemptible VMs are now up to 33% cheaper. Read the article for some tips and tricks.

• VM instance sizing recommendations is now in Beta. Look for that “recommendation” column in your GCE instance dashboard to save and optimize your resources.

• Regional Managed Instance Groups is a new feature now available in Beta. GCE can now automatically spread new VMs across zones in the same region equally for better availability. Check out this short one-minute video.

…in Dataflow :

• Enhanced Dataflow streaming pipelines management. At-least-once processing semantics with Drain.

• For all of you Pythonistas out there, support for Python on Cloud Dataflow is going beta.

• “Cloud Dataflow, Apache Beam, six months in“. Apache Beam code is now part of the Cloud Dataflow SDKs and you can “expect the SDK 2.x version-bump for Java to coincide with its first redistribution of Beam later this year”. Check out also this interview of Francis Perry on the Software Engineering Daily podcast.

…in BigQuery:

• In-memory query execution in Google BigQuery. Shuffling petabytes in memory, faster is obviously better.

• Going back a little further, here’s an amazing recap of recent BigQuery updates. Great explanation and lots of resources.

…in other GCP products:

• A new version of Cloud Datalab integrates with TensorFlow and allows to run locally. Local Datalab instances can still access BigQuery or Cloud Storage and a sample code for a machine learning model is included in the Datalab container.

• Mobile support for Stackdriver Error Reporting.

• Python 3 on App Engine flex is now in beta. This offers support for Python 3.4 and 2.7. Check out this one-minute video and read more on the Python developer hub.

• App Engine traffic splitting is an powerful yet very simple feature and there’s now documentation on how to leverage it right from your deployments. HTTP PATCH FTW.

• SSD Persistent Disks now offer 25,000 IOPS. Here’s what it means for your apps and how it compares to the competition.

 

GCP Podcast

No vacation either for the GCP Podcast with two episodes on Kubernetes 1.3:

• Part I on what’s new in Kubernetes 1.3.

• Part II on GKE.

The two most recent episodes were on SRE‘s (a fascinating conversation about how Google does operations) as well as an interview on the latest on PHP on GCP.

Other Interesting GCP News and tidbits

… in Machine Learning:

• TensorFlow in practice, a podcast interview with Rajat Monga.

• One of the key features of the Cloud Vision API is its SAFE_SEARCH_DETECTION. See how it can work for you (and for any site hosting user-generated content).

• Cloud Vision API powers Disney’s dragonspotting.com mobile website to find Elliot the Dragon in your surroundings! A really cool mobile web app too!

• Google uses DeepMind AI to cut data center energy bills.

• “Structuring unstructured text with the Google Cloud Natural Language API”. Fun with the Natural Language API, the Wikipedia API, AppEngine, Dataflow, and BigQuery.

… in big data:

• Building WePay‘s Webhook delivery system with Google Cloud Pub/Sub. Problem to solve, implementation, and lessons learned.

• Analysing Docker projects on Github with BigQuery.

• What a customer 10TB dataflow pipeline looks like.

… and on other general topics:

• Miles Ward, our Global Head of Solutions has couple of “run the same, everywhere” posts on why Cloud lock-in is bad and what we’re doing to keep our GCP customers away from it. Check out part I and part II. Hint: open source and community key ingredients.

• If you or your colleagues are seasoned AWS users, you’ll certainly find this updated “Google Cloud Platform for AWS Professionals” guide to be helpful.

• “OpenStack embraces Kubernetes to become a whole lot more like Google”

• “Orbitera joins the Google Cloud Platform team”

 

A VM is a VM is a VM (or is it?)

When I hear that all IaaS clouds were created equal I feel like a decade or so ago arguing that Solaris 10 was actually bring lots of innovation to the operating system market (Dtrace, ZFS, Zones). Google’s Compute Engine is adding unique features one at a time to the point where flexibility is really the real reason to move to the cloud.

It all starts with the ability to create your VM using custom machine types rather than having to chose from a pre-defined set of configurations. You can also change the characteristics of an existing machine – more memory, more CPU to adjust to different requirements or when moving from development to production. Then, the boot time has been repeatedly measured as being extremely fast which really relates to how flexible a cloud environment should be.

When running your VM, you can rely on live migration voodoo to keep your services running (while, for instance, Google upgrades the underlying infrastructure) as well as wonderful local SSD performance. Performance, reliability, flexibility. Choose three.

Price is always important and since you don’t have time to become a billing expert, Compute Engine is priced by the minute (so you truly pay for what you use), offers automatic discounts on sustained use and a crazy-cheap option with preemptible VMs. Customers of course benefit from the pricing competition and Google is committed to providing the best value.

Of course there are many other things that make Compute Engine attractive including Google’s networking infrastructure, the new UX-friendly Console with great in-browser SSH support, and of course all the higher-level services such as Container Engine (hosted Kubernetes). But at the end of the day what matters is that innovation and competition are alive and well.

Google works hard to make sure not all Cloud VMs are created equal but you should check out for yourself – you’re just minutes away from SSH’ing into your own Compute Engine VM.

Summer 2016 update: you can not also get sizing recommendations from Google to better optimize your instances, including those you should scale down!

Microservices still on the horizon

I don’t think everyone can (or should) jump on the microservices bandwagon just yet. If you can’t build a monolithic app, chances are you’ll face technical and organizational challenges trying to go full on with microservices.

if microservices is the goal then continuous integration and container orchestration are pretty much two required milestones on the journey towards achieving it.

Even if Continuous Integration isn’t yet implemented everywhere, especially in larger organizations, it’s now a pretty mature topic and most agile teams have now moved on to the even more rewarding continuousdeployment practice.

Container orchestration, on the other hand, is still too hard and definitely not yet a mature space. Just a year ago it took Mesos experts (or worse, roll-out-your-own-solution architects) to run a microservices architecture.

Only now with Kubernetes and other emerging solutions are we getting closer to a place where microservices can be conceived without the fear of having to implement the underlying resilient architecture to support them.

Containers vs Containers (in 2015)

2015 seems to be a good time to pooh-pooh Java Containers because Docker has obviously solved all the developer’s and ops’ problems. While bringing containers to the masses is a good thing ™, in server-side Java land you should still understand the distinction between “container” and “app server”.

The Java container provides services that adhere to an API, typically Java EE (with servlets being the most used API of all). The application server product, on the other hand, is the part that implements everything the API or the spec doesn’t require: management, monitoring, and of course clustering. This is where clearly, with the advent of Docker containers and the move to microservices, you should think seriously about the value this brings to your deployments, or rather the constraints it imposes.

But as far as the Java container is concerned, I’d say that it’s something that is still pretty much relevant. You may chose not to use all of the services the container has to offer (think JSF, EJB, JMS, …) but you might as well use the component models and separation of concerns it brings to the table.

This short piece was triggered by my morning read of James Ward’s excellent Comparing Application Deployment: 2005 vs. 2015. More on that in another post.

Google Cloud Platform LIVE – Décryptage

Si vous n’avez pas pu y assister, voici un résumé des annonces du Google Cloud Platform LIVE du 25 mars 2014.

Avant de rentrer dans les détails, pour ceux qui sont sur Paris, le Google Cloud Roadshow passera dans la capitale le 7 avril prochain chez Google France (inscriptions).

Urs?

La keynote (YouTube) était assurée par Urs Hölzle, le papa du Cloud de Google, interne comme externe. Pour le contexte, la veille, Urs a été interviewé par Wired. En résumé l’homme qui depuis le début est responsable de la conception des datacenters et du logiciel (GFS, MapReduce, BigTable, Colossus, Dremel, …) qui soutient le cloud le plus important au monde (Search, Maps, YouTube, GMail, …), nous promet un monde nouveau. Rien de moins. Voyons ce qu’il en est.

Pricing

Partant du constat que le pricing dans le cloud est trop complexe et trop cher (surtout par rapport au coût du hardware qui baisse avec la loi de Moore). Du coup, Google annonce des baisses de prix entre 30% et 85% avec l’engagement que les prix continueront à suivre celui de l’infrastructure sous-jacente (ces prix s’appliquent dans toutes les régions, historiquement l’Europe était un peu plus chère). Mieux encore, il n’est plus nécessaire de faire un choix entre instances normales et pré-réservées (moins chères): il suffit d’utiliser une instance au moins 25% du mois pour qu’un rabais supplémentaire de 30% s’applique!
Les détails de pricing sont ici

Nouveaux OS sur Compute Engine

En plus de Debian et CentOS, on compte parmi les images disponibles : RedHat Enterprise Linux et SUSE Linux Enterprise Server (dispo maintenant) et Windows Server 2008 R2 (en Preview pour le moment). Egalement en Preview, un nouveau service : Google Cloud DNS.

Big Data / Big Query

La récente fonctionnalité BigQuery Streaming permet désormais d’injecter 100 000 enregistrements par seconde! De quoi reconsidérer son warehousing ou son sharding.

Productivité du développeur

Trop souvent, dans le cloud, il faut choisir entre productivité (contraintes) et flexibilité (responsabilité). Les outils exposés aux développeurs sont aussi bien souvent des boites noires (“ne jamais laisser un développeur approcher des machines de prod!”). Dans un effort de simplification de l’expérience du développeur, le Cloud Google propose désormais :

• une intégration Git (GitHub ou un repo git privé et hébergé chez Google)
• la navigation depuis les messages d’erreur dans la console Cloud directement vers le code source
• l’édition en mode rapide du code toujours depuis la console
• le git-push-to-deploy avec build (Maven ou Gradle pour Java) et tests intégrés via jenkins et déploiement en production
• un SDK et sa ligne de commande gcloud qui unifie tous outils précédents, supporte des profils multiples et propose de l’auto-complétion sur toutes les plate-formes (Linux, Mac, Windows)

Détails dans ce blog en anglais.

IaaS ou PaaS?

On doit pouvoir faire mieux que la dichotomie classique entre IaaS et PaaS pour éviter un choix trop structurant en début de projet. C’est ce que promet Managed Virtual Machines, de quoi garder l’automatisme de App Engine et d’y intégrer des instances Compute Engine pour y exécuter du code natif ou carrément utiliser une stack logicielle comme node.js ou Dart. Puisqu’il s’agit de VMs Compute Engine, il reste tout à fait possible de s’y connecter en ssh. Pour autant ces machines sont provisionnées pour avoir accès de manière transparente à l’ensemble des autres services Cloud de Google (BigQuery, DataStore, Memcache, Cloud Storage, etc…).

Quand??

“Managed VM” propose un programme de Trusted Tester. Pour le reste, les nouveautés destinées aux développeurs (git, maven, jenkins, …) doivent apparaître dans quelques semaines.

Ces annonces ne sont qu’un début et synonyme de changements profonds internes chez Google. Prochaine étape majeure (enfin, après le roadshow) – Google I/O, fin Juin.

Alors, que pensez-vous de ce nouveau monde?

Docker on Google Compute Engine – a Hello World with OpenCV

Docker seems to strike a great middle-ground between complete-but-full-blown VM’s (a la VirtualBox or VMWare) and error-prone-but-super-lightweight setup/production scripts. Its architecture built on layers offers an interesting amount of granularity and I felt like I needed to build my own little Hello World.

I chose to build a Docker file for a build image [1] with OpenCV, a computer vision library, and specifically with its Java binding and test its deployment on Google Compute Engine [2]. While very popular, OpenCV isn’t particularly easy to set up and thus makes for a nice use-case for something not completely trivial as a random ‘apt-get install’. Note this is a trusted build, hosted on index.docker.io as a link to the Dockerfile on github and built by the Docker Index system.

---

After building a binary for OpenCV 2.4.7 together with its Java support (which is not part of the default build and somewhat tedious to generate), I created this simple image that simply places the compiled bits in /opt and places a simple face recognition demo application in the home directory. You’ll note that the build includes the install of an updated version of libc6-dev to satisfy Java OpenCV shared libraries. Docker nicely hides this mess to the user of the image.

Of course, there also needs to be a proper Java installed. This is done by building the OpenCV demo image from another trusted build (alexismp/debian-openjdk) which itself builds off of debian wheezy. As the name implies it installs OpenJDK (7) and configures JAVA_HOME and PATH.

Finally, my OpenCV demo image provisions ANT which is used to run the sample code with a trivial “ant build” command in the demo directory (/home/faceDetect).

Running this software stack on Google Compute Engine is as easy as pulling the build from Docker’s repository from a Docker-enabled virtual machine [1], running docker with shared directories, and moving the result to Google Cloud Storage :

$ sudo docker pull alexismp/opencv-demo-java7-trusted
$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
alexismp/opencv-demo-java7-trusted latest 4a9f88286d2e 36 hours ago 783.7 MB (virtual 5.123 GB)
$ sudo docker run \
    -v /tmp:/root \
    -t -i alexismp/opencv-demo-java7-trusted \
    bash
% cd home/faceDetect/
% ant
...
% cp *.png /root/
% exit

$ gsutil mb gs://face-detect/
$ gsutil cp /tmp/*.png gs://face-detect/

The OpenCV test uses face detection (not recognition! :) and creates a PNG images boxing the faces it has detected. Note how simple it is to use the Cloud Storage gsutil command-line to create a new bucket and copy the generated image which at this point is available from a RESTful API.

Being on the Google network with my Compute Engine instance made the pull operations pretty painless with great download speeds and the backup to Google Storage plain trivial. I also used a simple persistent disk with docker installed and configured which is booting in just a few seconds.

The next step is to integrate this functionality and make it available as a generic service, FDaaS, Face Detection as a Service! This will require writing a front-end or an offering an API and tweaking the Java code to deal with more flexible input/output.

---

Using Docker on the Mac currently required me to setup Vagrant, VirtualBox and Docker itself, something between dependency hell and Russian dolls but I’m told I should really look at boot2docker (which also sounds like an interesting solution for docker in production).

Interested in Docker? Their Getting Started page is pretty good. Start there and move to the equally good documentation when needed. Just remember that this is still a dot (0.x) release and officially not suited for production.

---

[1]: if you find the Docker vocabulary a bit confusing, you’re not alone. Check out this nicely written post : http://blog.thoward37.me/articles/where-are-docker-images-stored/

[2]: Docker on Google Compute Engine – http://docs.docker.io/en/master/installation/google/

Google presence at Devoxx Antwerp 2013

     
A busy week is about to start for the Devoxx 2013 participants and speakers (it’s been busy for organizers for many weeks!) and Google is back this year with a lot in store for developers.

Of course, you’ll hear a lot about the newly released Android 4.4 KitKat from Chet and Romain in a University session, a “What’s new” session, and more (Chet is bringing a tie…). Also covered will be Google Play Games Services with a hands-on lab (by Dan Galpin), an Android performance University session, an “image processing on Android” session by Nicolas Roard, a “Design” by Nick Butcher, and a puzzlers session!

AngularJS is back by popular demand with a lab, a university (both on Monday), a session, and a BOF session with a large portion of the team present at the conference (Igor, Misko, Brad, …). There’s also this related TDD session : “Unit testing JavaScript/CoffeeScript/Dart code with Karma”. The AngularJS BOF/Meetup is scheduled for Wednesday 7pm and I hear it’ll have fine food and drinks…

If you’re curious about how Chrome (Packaged Apps) can help you build rich clients there’s a full university waiting for you (by Jow Marini) and an NFC quickie by François Beaufort. Sam Dutton is back to present on Media APIs and WebRTC progress. Maybe in one of the most important Web sessions, Mike West will help you mitigate the risk of XSS.

The Google Cloud has grown to much more than just App Engine. Ludo will cover Google Compute Engine (infrastructure, VM-based offering), the many new features in App Engine, and how the two can seamlessly play together. Ludo will be busy with another session about how to build scalable RESTful APIs, and finally Martin Gorner with explain BigQuery for dummies (aka Dremel, yet another part of the Google’s Cloud).

If you run a web site or service with mobile users and you’re still in the business of managing your their passwords, I would recommend attending Ian Barber’s “Google Sign-in on mobile devices (Android and iOS)” session.

Finally, Thursday will be “Dart Day” for us starting with the Google keynote by Lars Bak, an industry veteran, author of the HotSpot JVM, of the V8 JavaScript engine and now leading the Dart effort, who will no doubt have a thing or two to say about the future of Web development. Later in the Day, Lars and the team will hold a “Dart – Ask us Anything” session during lunch, will a “Mobile, multi-device, multi-player with HTML5 and Dart” session in the afternoon. The day will end with a 6pm to 8pm happy hours with the Dart team (co-founders Lars Bak and Kasper Lund, Seth Ladd, and a number of other engineers). Make sure to stop by the Google booth early to get your free ticket.

Interviews, Booth, Cloud Credits, Quizz & More!
Similar to last year we’ll be conducting a number of interviews of Google speakers which will later be made available on the Google Developer YouTube channel as well as on Parleys. You are encouraged to ask questions ahead of time using this Moderator link.

We’re also running a Google+ event for Devoxx to share photos, comments and interact with other conference participants.

Devoxx participants, make sure you look for the Google Cloud Credits coupon in your attendee bag and redeem it! Also make sure you try the Google quizz online or at the Google booth, we have a number of nice giveaways for those that do!

And, of course, let’s meet at the Google booth and chat about anything developer!

Devoxx, here we come!