Three weeks in the life of Google Cloud Platform (GCP)

I was off on vacation with family for three weeks in August and came back to work to an impressive list of GCP (Google Cloud Platform) news and I thought I’d compile them here:

Product Launches & Announcements

Lots of GA announcements (Generally Available, fully-supported):

• Cloud Shell is now GA, and still free. \o/

• Cloud Datastore v1 API now GA. Full independence from App Engine if you want it! This release includes console enhancement, best practices, storage size calculation, clearer limits, and multi-tenancy guidance via namespaces. Also, it is now serving 15 trillion queries per month. Snapchat, Workiva and Khan Academy are among customers that can be named at this point.

• Google Cloud Bigtable is now GA. This is the fully-managed database service built on Google’s internal Bigtable service with Apache HBase client compatibility. It powers Google Search, Analytics, Maps, Gmail and more. Dataflow, Dataproc, Pub/Sub and BigQuery are all well integrated with Bigtable and Spotify, FIS and Energyworx are all using it already.

• Cloud SQL Second Generation is now GA. It offers MySQL 5.7, point-in-time-recovery (PITR), automatic storage re-sizing and failover replicas setup with a single click. It’s 7x faster and has 20x more storage capacity than its predecessor. See also the post for competitive benchmarks. Here are short videos on how to use Cloud SQL with App Engine and with Compute engine.

• Customer-Supplied Encryption Keys for Compute Engine goes GA. Get both control over data AND agility of the cloud. Now available in 14 countries.

The Cloud team is also exploring some (relatively) new and important territory :

• “Making ASP.NET apps first-class citizens on Google Cloud Platform”. Showing Microsoft developers lots of love with Windows Server on GCE, ASP.NET image, SQL Server on GCE, Google services via NuGet, Cloud Tools for Visual Studio, Cloud Tools for PowerShell and ASP.NET Core support coming soon. Check out this short one-minute intro video.

Some important product enhancements…

… in Compute Engine :

• Preemptible VMs are now up to 33% cheaper. Read the article for some tips and tricks.

• VM instance sizing recommendations is now in Beta. Look for that “recommendation” column in your GCE instance dashboard to save and optimize your resources.

• Regional Managed Instance Groups is a new feature now available in Beta. GCE can now automatically spread new VMs across zones in the same region equally for better availability. Check out this short one-minute video.

…in Dataflow :

• Enhanced Dataflow streaming pipelines management. At-least-once processing semantics with Drain.

• For all of you Pythonistas out there, support for Python on Cloud Dataflow is going beta.

• “Cloud Dataflow, Apache Beam, six months in“. Apache Beam code is now part of the Cloud Dataflow SDKs and you can “expect the SDK 2.x version-bump for Java to coincide with its first redistribution of Beam later this year”. Check out also this interview of Francis Perry on the Software Engineering Daily podcast.

…in BigQuery:

• In-memory query execution in Google BigQuery. Shuffling petabytes in memory, faster is obviously better.

• Going back a little further, here’s an amazing recap of recent BigQuery updates. Great explanation and lots of resources.

…in other GCP products:

• A new version of Cloud Datalab integrates with TensorFlow and allows to run locally. Local Datalab instances can still access BigQuery or Cloud Storage and a sample code for a machine learning model is included in the Datalab container.

• Mobile support for Stackdriver Error Reporting.

• Python 3 on App Engine flex is now in beta. This offers support for Python 3.4 and 2.7. Check out this one-minute video and read more on the Python developer hub.

• App Engine traffic splitting is an powerful yet very simple feature and there’s now documentation on how to leverage it right from your deployments. HTTP PATCH FTW.

• SSD Persistent Disks now offer 25,000 IOPS. Here’s what it means for your apps and how it compares to the competition.

 

GCP Podcast

No vacation either for the GCP Podcast with two episodes on Kubernetes 1.3:

• Part I on what’s new in Kubernetes 1.3.

• Part II on GKE.

The two most recent episodes were on SRE‘s (a fascinating conversation about how Google does operations) as well as an interview on the latest on PHP on GCP.

Other Interesting GCP News and tidbits

… in Machine Learning:

• TensorFlow in practice, a podcast interview with Rajat Monga.

• One of the key features of the Cloud Vision API is its SAFE_SEARCH_DETECTION. See how it can work for you (and for any site hosting user-generated content).

• Cloud Vision API powers Disney’s dragonspotting.com mobile website to find Elliot the Dragon in your surroundings! A really cool mobile web app too!

• Google uses DeepMind AI to cut data center energy bills.

• “Structuring unstructured text with the Google Cloud Natural Language API”. Fun with the Natural Language API, the Wikipedia API, AppEngine, Dataflow, and BigQuery.

… in big data:

• Building WePay‘s Webhook delivery system with Google Cloud Pub/Sub. Problem to solve, implementation, and lessons learned.

• Analysing Docker projects on Github with BigQuery.

• What a customer 10TB dataflow pipeline looks like.

… and on other general topics:

• Miles Ward, our Global Head of Solutions has couple of “run the same, everywhere” posts on why Cloud lock-in is bad and what we’re doing to keep our GCP customers away from it. Check out part I and part II. Hint: open source and community key ingredients.

• If you or your colleagues are seasoned AWS users, you’ll certainly find this updated “Google Cloud Platform for AWS Professionals” guide to be helpful.

• “OpenStack embraces Kubernetes to become a whole lot more like Google”

• “Orbitera joins the Google Cloud Platform team”