Forums  > Careers  > puppet, etc.  
Page 1 of 1
Display using:  


Total Posts: 206
Joined: Mar 2011
Posted: 2019-05-17 22:57
per the General thread on leaving [having left] finance.

I was talking to a former Murray Hill resident about the changes in server admin. Server admins of his vintage have been outmoded. Something similar to the Rails productivity revolution occurred within the last 5–10 years in server admin, which is why Puppet is now (seemingly) a ≥$100k skill.

Unlike "data science" or anything that claims to involve mathematics, increased productivity in terms of servers managed per person [puppet etc] or speed of website development per programmer [rails etc] actually makes sense.

And doesn’t depend upon knowing, or helping, oil billionaires.

For students, career changers, etc., with a tech bent, this might be a good direction.

I did the puppet learning VM. It hasn’t lead anywhere, but if I were 25 I would be looking into this kind of thing.


Total Posts: 10
Joined: Jul 2017
Posted: 2019-06-23 12:18
Puppet is (hopefully, in my case) a >$100k skill because it has not aged well, yet some businesses are tied to it. In the case of a technology IT workers actually want to use, you will observe the Haskell tax: salaries noticeably lower for the same job, but you don't want to shoot yourself at the end of the workday.

Remember that configuration management is solution to a problem that should not exist in the first place: virtual machines running software configured by silly little text files. OS processes were intended as virtual machines in the first place, and if at some point were deemed unfit to the spec, should have been fixed. In the same way, representing trees of labels in text files is utter silliness when your file system is a (potentially distributed) database optimised for exactly this use. PureFTPd is one of the few pieces of software were the designer demonstrates acceptance of this insight.

The new hype is containers anyway. My favorite question to docker-heads: what steps exactly does a sysadmin take to fix yet another OpenSSL vulnerability across his infrastructure? How is that less complicated than the old way?

Young, smart people should start with distributed algorithms so they can smell the bullshit from afar.


Total Posts: 45
Joined: Jul 2018
Posted: 2019-06-23 22:09
Your friend is mostly right, bare metal server admins are no longer worth much. Most data centers provide those services now; if not, all of them have resellers who have outsourced IT services that will act like a part of your team, have root access and keys. Many firms will bootstrap their internal infrastructure on AWS/GCE nowadays which further reduces their need for a bare metal sysadmin.

With this environment in mind, devops skills are more important than bare metal skills.

Puppet itself is not a $100k skill, but it could get you a foot in the door. You'll need a whole suite of skills like config management (Ansible, Puppet), CI (GitLab, Circle), containerization (Packer, Docker), log management (ELK), orchestration (Mesos, k8) - a lot of which are driven by cloud.

What's very rare to find is someone with some subset of these skills and finance, which is a specific blend of both very low level and high level knowledge. This could mean familiarity with Arista switches, ToE NICs, kernel source, prop feeds, HPC middleware, dashboards, instruction sets; someone with good 3 character memory, and so on.

Your friend's view is more up to date in finance, but even finance is half a step behind tech. Devops is also becoming quite commoditized in tech firms because there's a *aaS mentality as cloud providers and their ecosystem partners build more services on top. There are services that will deploy all of the devops best practices as a bundle for you at once nowadays. The mentality among 25 year olds at FAANG is that no one wants to be on pager duty for devops tasks, which is quite a shame, being on pager duty is a great way to learn how to architect distributed systems.


Total Posts: 45
Joined: Jul 2018
Posted: 2019-06-23 22:27
@sigterm There's many things that are hyped but I don't consider containerization one of them.

I can see your point in that I see trading firms forcefully adopting containerization in places where there's no meaningful advantage, just because of fear of missing out on what tech firms are doing. But containerization makes a lot of sense for a web app that needs to scale and have almost no downtime.

The reason is that an immutable configuration paradigm is so much easier to work with than a mutable one. This applies too to finance once you grow outside of 1 rack of footprint or have a decent trading strategy with several tiers of configs and meta-configs, and modeling results can be difficult to replicate because of config management issues. Especially with virtualization, it's far easier to handle any configuration change as the deployment of a new server or container than an update of current state.


Total Posts: 10
Joined: Jul 2017
Posted: 2019-06-24 10:43
Full disclosure: I think anyone pushing "devops" or the word "agile" as a noun should be chased out of the office with a broom. I accept "SRE" as a shibboleth for people who do what devops do but may also possess a brain.

In my immediate surroundings, I know of many tries to use containers (insurance and telecoms mostly), but only one serious success, using Rancher (the first one, *not* the Kubernetes one, which they've tried too). Other minor instances using Docker. Ask any friend working at Google if they use Kubernetes themselves: no, they keep to Borg; guess they know better than to taste their own poison.

Immutability is indeed the way to make all of this work much better, but I have not seen the idea penetrate my environment. Must have not made it into the Gartner quadrant. Also, I would consider Erlang as a proper, mature implementation of this idea (additionally subsuming the concepts of microservices/JSON/REST and containers into the language VM, with little configuration worry), yet it's completely ignored. No love for NixOS either it seems. SmallTalk had stateful objects moving live between computers in a network, so that was solved long ago too.

As for scaling out stateless code like web applications, it is the easiest problem in the whole infrastructure, compared especially to the management of stateful processes and communication -- anywhere where side-effects hide, basically. Yet it is a problem that receives enormous attention, with extremely ornate solutions to it, while reliability is an afterthought and observability is thrown out the window. Tells you a lot about the culture behind it.


Total Posts: 322
Joined: Feb 2014
Posted: 2019-06-24 14:11
containerization make a lot of sense - companies are doing that because it's cheaper and simpler - we saved a lot of opex $ from servers and licenses; you cant call it hype if this solving real problem; &&


Total Posts: 45
Joined: Jul 2018
Posted: 2019-06-24 23:48
I'm not partial towards any particular job title. SRE is OK but many recruiters will mistake that for an emphasis on the systems support, which I still consider more 'ops' than 'dev' work.

Every trading platform I know in production today pushing over 1/10 ADV in equities, options, futures, FX has horrible uptime compared to that of a single SSD or data center SLA, but amazing recovery. Systems downtime ('ops' issue) isn't as frightening as it sounds even if you're pushing over 10% ADV. Sure, you're leaving good upside on the table, but chances are that your orders are/can be pulled almost instantly and you just have a large NOP. Recovery is a much more meaningful issue for these cases, and lies more with the application design ('dev' issue) than hardware-level persistence.

One specific example that comes to mind was KRX's KOSPI tick size change in Mar 2017, which was quite the mess. Here's a critical change with intraday effects unceremoniously announced on a non-machine readable site. A useful devops person should understand how to deploy reference data and strategy version changes to handle any related intraday conversions as well as trading suspensions in other KOSPI 200 derivatives like FMK2 on Eurex. I'm not sure if that's what's entailed in your idea of a brainy 'SRE', but definitely what I expect of a good devops person.

I've never worked in an 'agile' environment so I have no input there.


Total Posts: 1161
Joined: Jun 2007
Posted: 2019-06-25 06:52
Well...."agile" is a completely new topic. I have to admit even though it sucks at times, it isn't the worst way to work. I feel it isn't that great for projects that involve a lot of research....don't want run into the cliche of the hippster team that runs a big web application, but I still think agile methods work best in these kind of projects.

Regarding containers: we use docker a lot and as stack said it can add a lot of value. I think that probably to many teams run amok and go the whole way with kubernetes and shit even though it is not necessary.

What bugs me more....closely connected to the docker stuff are micro-service architectures and CD paradigm. And here my experience is that everybody tries to be NETFLIX, even though they are most definitely not. The problem is most C-suits don't get THAT THEY WILL NEVER GET RID OF THE BLOATED TEAMS (one team per service).....quasi by design. They think they can be super agile until the product is "finished"....and then hand it to a small ops team. The microservice approach is used in cases where you don't have to reiterate and deploy several times a day.

Sorry for the rant

Ich kam hierher und sah dich und deine Leute lächeln, und sagte mir: Maggette, scheiss auf den small talk, lass lieber deine Fäuste sprechen...
Previous Thread :: Next Thread 
Page 1 of 1