I'm *still* doing this. Set up OpenTelemetry on a Phoenix project a day or two after I started it, set it up to send events from my local workstation into Honeycomb.

Learned a ton about Elixir this way -- how releases and config work especially. Also came in handy when I was getting exceptions back from Postgres because of bad validation.
https://www.simplermachines.com/why-i-set-up-honeycomb-on-small-projects/

@nat I’ve really got to try out Honeycomb sometime. It’s high on my list of services I want to explore!

@alpha Highly recommend it! I also hear good things about Lightstep.

Assuming the project I'm working on right now gets traction I'm going to explore using it to replace integration/acceptance tests, and integrating it more into a test-first workflow.

@alpha Happy to get on a pairing session anytime if you ever need help debugging a telemetry issue too. This is an area where I'm actively investing being able to do the basic stuff fast in any language.

@nat Will definitely take you up on that if I’ve ever in that position - currently don’t have any side projects, so don’t really have any problems for telemetry to solve!

Though actually, in Q2, our platform team will actually start working on observability in earnest, and some of us had been wondering where Honeycomb fits vs. Datadog, etc. Do you have opinions/recommendations/resources on that? My knowledge is almost a decade out of date at this point (though IMO, our ELK + statsd setup was pretty damn fine back then and better than a lot of the things I’ve used on Labs engagements, though that’s a pretty low bar).

@alpha The Honeycomb vs. Datadog choice basically hinges on three things.

1) Pricing models. Honeycomb charges per event, Datadog charges per host.
2) Tags/attributes. Datadog has a hard limit on how many attributes you can stick on anything. You have to go in and explicitly set things you want to be able to search on to be searchable in their UI. Whereas with Honeycomb being able to put arbitrary numbers of attributes on things and then search on them is the whole point.

@alpha
3) Overall integration. Do you need one vendor who can do everything Datadog can do? Will you invest in actually using everything Datadog does? If the answers to both questions are "yes" you'll get a lot more value out of Datadog than if you're only using a slice of its capabilities or you don't invest in a full integration.

@alpha Some other factors include --

How mature is the OpenTelemetry integration for the stack you're using? If you're using something that Datadog's agents handle better than Otel that might be a reason to use Datadog.

If you're in something that Otel handles really well, though, you can do a generic Otel integration, send your stats to all the vendors you're considering, and do a head-to-head comparison with live data.

@nat Ugh, what this tells me is that we really need to hire a devops person.

@alpha An important fact though is that while Datadog claims that they can do what Honeycomb does technically but they fundamentally can't. Their underlying data model is fundamentally structured logs and they *cannot* do the fast searches on large datasets that enables Honeycomb's exploratory querying. It's possible they're working on this and I am not up-to-date but it'd be a huge lift.

See the Honeycomb article about their columnar data store for the details.

https://www.honeycomb.io/blog/why-observability-requires-distributed-column-store

@nat TBH, for our purposes, just having our metrics and logging centralized in something that’s not SQL will already be a huge win.

@alpha Probably but just something to keep in mind is that if what you need is ~1-3 months of a devops person, I am available on a contract basis and *love* doing this kind of work.

@nat Yeessss, let me suggest that to my leadership.

@alpha *checks where you work*

Oh hey wow I am extremely the target audience for this.