Tom Ron

One Gateway to Route Them All

Sat, 29 Aug 2026 20:03:23 GMT

In August, Stripe announced it had agreed to acquire OpenRouter — reportedly for more than $7 billion, per Bloomberg. OpenRouter helps businesses route and optimize token usage across 400+ models from more than 80 providers, and it fits alongside the token-billing work Stripe has been shipping for a year. Earlier in 2026, Palo Alto Networks bought Portkey to anchor its Prisma AIRS platform.

The layer in question is the AI gateway: a single service that sits between your application and every model provider you call. Your code talks to the gateway; the gateway talks to OpenAI, Anthropic, Google, Bedrock, or whatever open-weight model you're hosting yourself. If your team calls more than one provider, you've probably built some version of this already, even if you didn't call it that. It usually starts as a thin wrapper around an API client, then grows into something bigger once you realize you can't answer basic questions like "how much did we spend on GPT-5 last week" or "what happens when Anthropic has an outage." A gateway turns that ad-hoc wrapper into infrastructure. Four reasons it's worth doing deliberately:

Unified access - One API, one auth scheme, many models. Instead of maintaining separate SDKs and error-handling paths for OpenAI, Anthropic, Google, and whatever open-weight model your ML team is experimenting with this month, you integrate once and swap models by changing a string - instead of migrating twelve call sites when a provider deprecates an endpoint.
Cost observability and savings - The gateway sits in the one place that sees every request, making it the natural home for spend attribution by team, project, or feature - and for caching, routing to cheaper models, and budget enforcement.
Compliance - Once you have that central choke point, you can enforce data residency, redact PII before it leaves your network, log everything for audit, and apply guardrails consistently instead of per-integration.
Business continuity - the gateway already knows about every provider; it's the natural place to build fallbacks - if Anthropic degrades, silently retry on OpenAI or a self-hosted model, instead of your product going down with the provider.

Which products are out there?

OpenRouter -The household name: a hosted marketplace fronting 400+ models from 80+ providers, with automatic routing between providers serving the same model when one degrades. Breadth and zero setup are the pitch — one API key, immediate access to essentially everything worth calling, per-provider fallback included. Pricing is pass-through on tokens with a flat fee on credit purchases, around 5.5% on card, cheaper via BYOK. Interesting to watch whether the fee or business model changes once it becomes a Stripe subsidiary.

LiteLLM - The opposite bet: an open-source proxy you run yourself, OpenAI-compatible across 100+ providers, no vendor in your request path and no markup — you pay providers directly. You get control: your VPC, your logs, your keys, free core routing and failover. You also get the ops - the Kubernetes deployment, the database behind spend tracking, etc. Open-core pricing: free to self-host; Enterprise adds SSO, RBAC, and audit logs.

Cloudflare AI Gateway - Rides the edge network that already terminates much of the internet's traffic, so the pitch is proximity and operational simplicity rather than model breadth: caching, rate limiting, logging, and guardrails close to your users, obtained by pointing your existing SDK at a new base URL. Dashboard analytics, caching, and rate limiting are free on all plans; persistent logs are across every tier; and it now supports OpenAI, Anthropic, and Responses-compatible endpoints. No markup on inference - a 5% fee applies only if you route third-party model spend through Cloudflare's Unified Billing instead of bringing your own keys. Strongest if you're already on Cloudflare; weakest if you need the deepest guardrail and compliance tooling.

Portkey - gateway as security control plane. Routing was never the whole product = observability, a prompt store, and governance guardrails came bundled, aimed at enterprises that need policy enforcement as much as uptime. That's what Palo Alto Networks bought in May, folding it in as the foundational gateway under Prisma AIRS, on the argument that once agents act autonomously, the gateway is the last place you can inspect a bad decision before it executes. Pricing is usage-based on logged requests, with retention gated by tier. Built from the ground up for governing what agents are allowed to do, not just which model answers.

Bifrost - an open-source gateway written in Go by Maxim AI, unifying 1,000+ models across 23+ providers behind one OpenAI-compatible API. Benchmarked at roughly 11 microseconds of overhead per request at 5,000 RPS. Covers the same ground as the rest - automatic failover, adaptive load balancing across keys and providers, semantic caching, hierarchical budgets, native MCP for agentic tool use - and plugs into Maxim's evaluation and observability platform. Open-core: OSS gateway free and self-hosted; Enterprise adds private networking, security integrations, and support.

If you don't have a gateway yet, you'll probably want one soon. What pushes you there varies - a finance question, an outage, a compliance review, etc. Whichever reason gets you to build it, you tend to pick up the rest on the way, because they all live in the same place: once every request passes through one service, spend attribution, failover, and PII redaction stop being separate projects. And as with everything else in software, what you should protect is your ability to leave. The gateway exists to keep you from being locked into a single model provider, so don't trade it for getting locked into the gateway instead. Ideally, switching from one to another is a base URL change and nothing more.

Lead with Context

Mon, 20 Jul 2026 14:18:41 GMT

In recent weeks, I completed Northwestern University’s Organizational Leadership Specialization on Coursera. While the courses span different topics- storytelling, social influence, negotiation, and more- two recurring themes stood out to me.

1. Preparation changes everything
Effective leadership and communication rarely happen “in the moment.” Preparation matters - even when you don’t yet know the formal frameworks or terminology.

Whether it’s thinking through your BATNA (Best Alternative to a Negotiated Agreement), identifying your ZOPA (Zone of Possible Agreement), or simply anticipating objections, the key habit is stepping into the other side’s perspective before the interaction happens. Strong outcomes are usually built long before the conversation begins.

2. The audience is the message
People are driven by different motivations, constraints, and perceptions. That means the same message can land very differently depending on how - and to whom - it is delivered.

Leadership is not just about what you say, but how you tailor it: language, framing, medium, and emphasis all matter. This is something I first learned as a scout leader, and it has stayed with me for nearly 25 years. It remains one of the most consistently useful leadership principles I’ve encountered.

At the same time, I’ve often seen a common anti-pattern: leaders trying to motivate others based on what motivates them, rather than what actually drives the person in front of them. Good intent, misaligned execution.

Ultimately, both lessons point to the same idea: leadership is less about projecting your intent and more about deeply understanding context - both the situation and the people in it.

Old leadership lessons, new context

Thu, 09 Jul 2026 17:16:45 GMT

Ron Wallace started as a UPS driver in rural Idaho in 1966 and retired 36 years later as President of UPS International, running operations in 200+ countries. His book, "Leadership Lessons from a UPS Driver," turns that climb into a handful of blunt, unglamorous principles. Here are four insights that stayed with me -

1. Best 5 vs 5 best.

Five average players who move as a unit beat five stars who don't. With AI blurring roles - PMs writing code, engineers writing specs - how well a team moves together matters more than any one person's skill ceiling.

2. Manage yourself before you manage others.

You can't ask for discipline you don't have. When everyone has an agent doing part of their job, self-management - what to delegate, what to check, what to own - is the whole game.

3. 10% planning, 90% execution.

That ratio assumed execution was the bottleneck. Agents and other tools now handle much of the execution. Does the split flip? Not sure - bad planning just produces fast, wrong output at scale. But the balance is shifting, and finding the new ratio is one of the more interesting problems before us.

4. Change is inevitable; growth is optional.

UPS survived by repeatedly outgrowing its own identity. Feels like the line of the year - nobody's job description is stable, and adapting is a choice, not a default.

Token Saving & Compression: A Practitioner's Review

Mon, 06 Jul 2026 20:34:15 GMT

AI coding agents burn tokens on things they don't need to see in full - verbose logs, entire files read for one function, repeated context across turns. A wave of tools now attacks this from different angles: some rewrite noisy output before it reaches the model, some compress what's already in context, some change what the agent generates or reads in the first place. Here is a short survey about the different tools.

rtk — github.com/rtk-ai/rtk
A CLI proxy that hooks into Bash tool calls and rewrites common commands (git status, cargo test, kubectl, etc.) into compact equivalents before output hits context - a noisy git push becomes ok main. Claims 60–90% savings on covered operations, plausible since the approach is narrow and mechanical.
Watch for: only intercepts Bash calls - native Read/Grep tool usage bypasses it entirely, so real-world savings depend heavily on how your agent works. Best for: teams whose agents lean on shell commands for everything (git, test runners, infra CLIs); less useful if the agent mostly reads files directly.

Caveman — getcaveman.dev
A family of products: a compression skill, a standalone token-budgeted coding agent, a persistent memory layer, and a compression proxy (beta). Claims ~65% token reduction.
Watch for: thin evidence - the site leans on stars and HN placement rather than a reproducible benchmark; the four products make it unclear which piece drives the number. Best for: teams wanting a single umbrella tool across several angles at once, and willing to validate the number themselves before committing.

Ponytail — alphamatch.ai/blog/ponytail-ai-coding-skill-2026
Attacks generation, not context: forces the model through a "does this need to exist" ladder before writing code, preferring stdlib/native/existing-dependency solutions. Own benchmark shows LOC cut to ~46% of baseline and cost down by 47–77%, but token usage itself drops only by ~16%.
Watch for: it's solving a different problem than the others — don't expect it to shrink context windows; the win is fewer lines and less rework, not less reading. Best for: codebases with a lot of unnecessary custom code; less relevant if your token spend is mostly context/tool output, not generation.

Headroom — headroom-docs.vercel.app/docs
Compresses everything an agent reads — tool output, file reads, API/DB responses — via type-specific compressors, with a reversible retrieve path back to full detail. Vendor case study: 87.6% fewer input tokens in a log-heavy scenario.
Watch for: numbers are vendor case studies on scenarios they chose; the retrieval mechanism adds a round trip when the model needs detail back. Best for: agents that use tools heavily against verbose sources (logs, API responses, DB queries); less relevant for short, code-only sessions.

Graphify — github.com/Graphify-Labs/graphify
Avoids re-reading altogether: builds a local knowledge graph of the codebase, then the agent queries it instead of repeatedly grepping files.
Watch for: it's not compression — savings materialize only over a session with repeated lookups in a large, stable codebase; non-code content (docs, PDFs) still requires API calls to extract. Best for: large, mature repos with frequent navigation; little benefit for small projects or one-off tasks.

Tool	Mechanism	What it targets	Claimed savings	Evidence
rtk	Rewrites known-noisy CLI output	Bash tool output only	60–90% on covered ops	Vendor, narrow scope
Caveman	Compression skill + memory layer + proxy	Prompts, outputs, cross-turn context	~65% tokens	Vendor, thin (stars/HN)
Ponytail	Decision ladder before writing code	Code generation, not context	~16% tokens; 47–77% cost; 3–6x speed	Vendor benchmark (5 tasks × 3 models)
Headroom	Type-aware compression + reversible cache	All tool/context input	87.6% input tokens (case study)	Vendor case study
Graphify	Knowledge graph instead of re-reading files	Repeated file reads/greps	Not quantified	Vendor, mechanism differs from compression

I want more. Can we stack them on one another?

Each method targets different layers, so most combinations aren't mutually exclusive. Ponytail acts at generation time (what code gets written), Graphify acts at retrieval time (avoiding re-reads), and rtk/Caveman/Headroom all act at the context-shaping layer, i.e., what enters the prompt. In principle, we can run all method at once without them fighting each other structurally.

Where it gets tricky is the overlap among the context-shaping tools. rtk and Headroom both intercept and rewrite content before it reaches the model, just at different points - rtk hooks Bash tool calls while Headroom sits more generally across tool outputs, file reads, and API responses. Stacked, we're likely to get diminishing returns rather than additive savings: once rtk has already collapsed a git status to one line, there's little left for Headroom to compress on that same output. Caveman's compression skill/proxy likely overlaps with both in the same way—three tools converging on the same noisy CLI output mostly means redundant passes, not three times the savings. Worse case: extra latency for near-zero extra token reduction.

Ponytail and Graphify are the cleaner additions. Ponytail's effect is upstream of everything else (less code gets written and read in the first place), so it should compound with anything downstream rather than compete with it. Graphify's effect is about not re-fetching content at all - if the agent never re-reads a file, there's nothing left for rtk/Caveman/Headroom to compress on that read, which sidesteps the overlap problem entirely rather than creating one.

The more promising stack is probably Ponytail (write less) + Graphify (re-read less) + one of rtk/Caveman/Headroom (compress what's left) — rather than running all three context-shaping tools together, which likely adds overhead with little additional token savings.

Reducing token cost is an issue every firm currently deals with, and it isn't one problem - it's at least three: noisy tool output, unnecessary code generation, and repeated re-reading of the same content. rtk, Caveman, and Headroom all attack the first at different points in the pipeline, with real but overlapping coverage. Ponytail attacks the second, upstream of everything else. Graphify sidesteps the third by building a queryable map instead of re-fetching files. None of these are competing solutions to the same problem so much as partial answers to different pieces of it - which means the right setup is probably a small combination, not a single winner, and probably depends more on where your own token spend actually goes (shell noise vs. generated code vs. repeat lookups) than on any vendor's headline number. All of the figures cited here are self-reported, and the stacking behavior is reasoned from mechanism rather than measured - so treat this as a map of what to test, not a verdict on what to use.

Spec-Driven Development

Tue, 12 May 2026 06:59:05 GMT

I recently read Spec-Driven Development by Haim Michale, and it resonated with some day-to-day thoughts about SDD.

The problem SDD aims to solve isn't new

Missing, incomplete, and ambiguous requirements that evolve over time have always plagued software delivery, but it is now amplified: the pace is faster, LLMs have a strong tendency to fill gaps by guessing, and they carry zero tribal knowledge - no kitchen-table conversations, no "oh, we tried that in 2022" instincts. EARS syntax (Easy Approach to Requirements Syntax) is an interesting framework for making requirements more machine-readable, but it still requires a human to steer it. The spec is only as good as the intent behind it.

Legacy code and brownfield projects are the elephant in the room

Most SDD content assumes you're starting fresh. But the vast majority of engineering work happens in systems with years of accumulated decisions, undocumented assumptions, and implicit context baked into the codebase. Making that accessible to coding agents, let alone to SDD, is a real and largely unsolved problem. As far as I'm aware, there are no clear best practices or guidelines here yet. This is an area we need to tackle head-on.

SDD also accelerates a role blur we're already talking about

If specs are first-class artifacts that live in the repo, who writes them? Do PMs start pushing requirements documents directly into codebases, expanding into territory traditionally owned by engineers? Do developers take on deeper responsibility for refining and maintaining specifications? Both are plausible. Neither is cost-free. The boundary is getting blurry fast.

And drift…

Drift is what happens when specs and code silently diverge: a requirement changes, but the spec file doesn't, or a PR lands and the corresponding document isn't updated. Over time, specs stop describing the system, and that's arguably worse than having no specs at all, because they create false confidence. My suggestion: drift detection should be treated as a core part of repo gardening. Flag when code changes without a corresponding spec update, and vice versa. Make it visible, not something that accumulates quietly in the background. Expect spec debt terminology coming soon.

While different SDD methodologies vary in the exact documents they produce, I found the attached screenshot a useful cheat sheet for knowing when to update which document.

RAG Made Simple

Mon, 11 May 2026 07:35:12 GMT

📚 Just finished "RAG Made Simple: The Complete Visual Guide to Retrieval-Augmented Generation" by Nir Diamant, and it happened to be one of those rare cases where the book actually matches how I learn. What worked especially well for me are the summaries - when to use each technique, when it's overkill, and what the trade-offs are. That comparative lens - grounding new methods against what came before is exactly what I need to internalize something. If you only have time for a sneak peek, read the appendix on debugging and choosing the right approach.

Theory vs. practice - in real production systems, no single RAG method is enough. Different techniques address different failure modes, and a stable, reliable system almost always requires combining several of them.

Three techniques worth calling out:

🔮 HyDE (Hypothetical Document Embeddings) - my most surprising find. The idea is to generate a fake answer to the query, then use it to retrieve real documents. Elegant solution to the structural mismatch between how questions are phrased and how answers are written. "Fake it 'til you make it" as a retrieval strategy.

🎯 Dartboard Retrieval - most useful for me personally. A problem I've bumped into repeatedly - in RAG and in recommendation systems + where you need to balance precision with coverage across varying distance thresholds.

🖼️ Multi-modal RAG - most overlooked, and massively underrated, usually skipped when building an MVP. Especially for internal knowledge bases: being able to retrieve diagrams, tables, screenshots, and slides alongside text is genuinely transformative. In my experience, much of an organization's information is in decks, charts, images, etc.

RAG made Simple cover

Kill Skill

Wed, 18 Mar 2026 10:20:53 GMT

Three recent incidents that should make all of us pause:

→ A hallucinated npx command spread through a single LLM-generated skill file to 237 repositories. Real agents executed it. A researcher claimed the package name before an attacker could.

→ 230+ malicious skills uploaded to OpenClaw's ClawHub in days. The #1-ranked skill was silently exfiltrating data and injecting prompts to bypass safety guidelines. Thousands of downloads before anyone noticed.

→ An audit of 2,890+ OpenClaw skills found 41.7% contain serious security vulnerabilities.

This isn't a niche problem. If you use Claude Code, Cursor, Copilot, or any modern coding agent, you're likely installing multiple skills a week. They are recommended by colleagues, appear in blog posts, and are bundled into project templates. The install command is one line. The trust is implicit.

Skills blur the line between configuration and code, but we treat them like documentation. A SKILL.md file can contain natural-language instructions, executable scripts, and package dependencies. There's no clear boundary where "docs" ends and "code" begins. No lockfile. No integrity checks. No verified publisher identity. The skills CLI has a package-lock.json for its own dependencies - just not for the skills you install.

So what should we do today? At a minimum, read the skill files before installing. It takes 2 minutes and is the equivalent of reviewing a PR. Beyond that, copy and commit your skills folder to git - your repo becomes your lockfile. Treat any skill update like a dependency upgrade: review the diff before merging, also for testing and quality purposes.

The tools are coming - Longer term, this ecosystem needs what npm took 6 years to build: lockfiles, signed packages, verified publishers, and scanners that flag malicious instructions before they reach your agent. Cisco developed an open-source skill file scanner.

Links -

Snyk Finds Prompt Injection in 36%, 1467 Malicious Payloads in a ToxicSkills Study of Agent Skills Supply Chain Compromise - https://snyk.io/blog/toxicskills-malicious-ai-agent-skills-clawhub/

Update #40: Agent Skill Security Issues - https://maxcorbridge.substack.com/p/update-40-agent-skill-security-issues

Agent Skills Are Spreading Hallucinated npx Commands - https://www.aikido.dev/blog/agent-skills-spreading-hallucinated-npx-commands

Over 41% of Popular OpenClaw Skills Found to Contain Security Vulnerabilities - https://www.esecurityplanet.com/threats/over-41-of-popular-openclaw-skills-found-to-contain-se

Skill Scanner - https://github.com/cisco-ai-defense/skill-scanner

Make PRs Logical Again

Tue, 24 Feb 2026 08:52:45 GMT

The day-to-day work of software engineers is shifting - less time is spent on implementation, and more on specifying, planning, and reviewing. As AI coding tools improve, the bottleneck moves upstream and downstream: writing clear specs, structuring changes well, and conducting thoughtful reviews become the highest-leverage skills.

One surprisingly weak point in that workflow is GitHub PR file order. Files are shown in alphabetical order that rarely reflects the logical flow of the change. Good reviews are narrative: start with the contract, then the core logic, then the edges and tests. When the order is wrong, reviewers waste cognitive energy reconstructing the story instead of evaluating the change.

That realization pushed me to build a small Chrome extension that lets you reorder files in a PR so the review reads in the right order. Check it out here.

It was also a personal reminder of the joy of doing something for the first time - I never built a Chrome extension before. My first attempt was messy and unsuccessful, so I wiped everything and restarted using Spec-Driven Development (SDD) with OpenSpec. The second iteration was dramatically smoother, more structured, and consumed fewer tokens. In a world where implementation is getting cheaper, clarity in specs, understanding of trade-offs, and the review experience are quickly becoming the real craft of engineering.

7 Lessons I Learned from Rugby Sevens About Engineering Leadership

Mon, 16 Feb 2026 09:41:00 GMT

Fun fact about me - I played Rugby Sevens for years, including on a national team. It taught me more about engineering leadership than I expected. Small teams, huge space, high speed - Sevens and modern engineering run on the same dynamics.

Here are 7 lessons Sevens taught me about engineering leadership.

1️⃣ It’s Not About Weight - It’s About Teamwork

The first lesson I learned came from scrumming with a weight disadvantage but a huge coordination advantage. In Rugby (in 15s even more than in 7s), you don’t win scrums because your players are heavier. A team wins because: you bind together, you push at the same moment, you trust the rhythm. In lineouts, timing matters more than height, and coordination matters more than individual strength.

In engineering, this shows up everywhere:

A brilliant engineer misaligned with the team slows progress.
A “lighter” team moving in sync outperforms a group of heavy hitters pulling in different directions.

Velocity is not the sum of individual forces. It’s a synchronized force.

Lesson: Coordination multiplies talent.

https://youtu.be/9VlRCQCjQDU?si=fUxT2T-oINQpoEcE

2️⃣ Communication should be Constant and Loud

In Rugby Sevens, silence is dangerous. Players shout constantly:

“Inside!”
“Switch!”
“Up!”
“Blind!”

It’s not noise - it’s alignment. There’s too much space and speed to assume others know what you’re thinking.

In an office, you get accidental alignment:

Body language
Whiteboard moments
Corridor clarifications

In remote work, silence becomes ambiguity. If communication isn’t explicit and frequent, gaps appear.

Lesson: Constant communication creates alignment. Alignment brings results.

3️⃣ You Can Only Pass Backwards

In rugby, you’re only allowed to pass the ball backwards.

If you want to move forward, you must either run forward yourself or align with teammates who are already in motion

You can’t throw the ball ahead and hope someone figures it out. The constraint forces structure.

To gain ground, teams spread wide.
They create overlap.
They run support lines before they’re needed.
They time short, precise passes.

You go forward by passing backward - and that only works with discipline. In engineering, this principle shows up everywhere. You can’t “pass forward” sloppily:

Product can’t throw half-defined specs over the wall
Engineering can’t push messy code to QA
Leadership can’t announce strategy without alignment

When work is tossed ahead without positioning and support, the play breaks. Real velocity doesn’t come from heroic sprints. It comes from synchronized movement.

Too many handoffs? You lose momentum.
Too few? You get isolated and tackled.

Lesson: Sustainable forward progress requires structured alignment. Coordination creates speed.

4️⃣ Reset Between Games

In Sevens tournaments, you play multiple games on the same day.

Win big? Reset.
Lose badly? Reset.

The next kickoff has no memory.

Engineering teams often carry emotional baggage:

A big launch → complacency
An outage → overreaction

Strong teams don’t ignore outcomes - they process them quickly.
They celebrate briefly, learn fast, and show up focused for the next “game.”

Lesson: Don’t let yesterday’s result dictate today’s execution.

5️⃣ Scoring Under the Posts vs Securing the Try

In rugby, when a player breaks through, they often try to run closer to the center before grounding the ball.

Why? Because it makes the conversion kick easier.

But that extra effort increases the risk of being tackled and losing the try entirely.

In engineering:

Do we ship now?
Or optimize a bit more?
Refactor first?
Polish the UI further?

Sometimes we optimize the conversion and lose the try.

Lesson: Secure value first. Optimize second.

6️⃣ You Don’t Always Need More Resources - Just a Change of Angle

In Sevens, attacking the wide-open side is obvious. Great teams exploit the blind side instead.

You don’t need more force.
You need better perspective.

In engineering:

Reframing a product problem
Reorganizing teams instead of hiring more
Solving a process issue instead of pushing harder

Sometimes the breakthrough isn’t scaling effort - it’s shifting angle.

Lesson: Strategy beats brute force.

7️⃣ Commit Fast - Adjust Faster

In Rugby Sevens, hesitation kills.

Pause before a tackle, you miss.
Delay the pass, the overlap's gone.
Half-commit to a line break, and you get isolated.

The game rewards decisiveness.

Engineering leadership is the same. Over-analysis drains momentum. Once you pick a direction, the team needs full commitment - not tentative buy-in where everyone's still hedging.

But here's the nuance: when a defender overcommits, great players side-step. Decisiveness doesn't mean rigidity. You commit fully, and when new information shows up or the situation shifts, you adjust fast - without ego, without drama.

Lesson: Commit hard. Adjust harder.

5 interesting things (12/02/2026)

Thu, 12 Feb 2026 13:29:37 GMT

The Quarterback Paradox - while I'm not sure I agree that it is a paradox - i.e., recruiting a critical position to an organization is hard even if you have a lot of data, I love and strongly agree with the suffix of the post - "As in the NFL, in organizations the hardest part is often not finding talent, but creating the conditions in which real potential does not break before it has a chance to become reality."

https://noamwakrat.medium.com/the-quarterback-paradox-e93e4325bac1

What LEGO Can Teach Us about Autonomy and Engagement - Who doesn't like LEGO? We all played with it as children, and some of us still build today. In this post, Pawel Brodzinski describes a neat experiment he runs in training sessions - teams first build a LEGO set under a manager's direction, then self-organize for a second build, and consistently report higher engagement when given more autonomy. While it shows a clear effect, the experiment has some drawbacks - most notably an order effect: the self-organized build always comes second, so the engagement boost could partly stem from participants being warmed up and more comfortable rather than from autonomy alone. Always nice to read about LEGO as an adult.

https://brodzinski.com/2026/01/lego-autonomy-engagement.html

Skyll - Skills are markdown instruction files that teach AI coding agents how to perform specific tasks. Today, skills must be manually installed before a session, meaning developers need to know upfront which skills they'll need. Skyll is an open-source search engine and API that lets any AI agent discover and retrieve skills on demand at runtime, ranked by relevance, without pre-installation. You can think of it as a package manager for agent capabilities, enabling agents to be truly self-extending and autonomous.

https://github.com/assafelovic/skyll

Skyhook.io radar - Existing K8s dashboards tend to be either heavyweight, cloud-dependent, or require cluster-side components. Radar's zero-install, single-binary approach with real-time topology and traffic visualization answers the need of developers and platform teams who want quick, frictionless cluster observability that can even run on their laptop, especially useful for DevEx-focused teams looking to reduce the friction of Kubernetes debugging and operations,

https://github.com/skyhook-io/radar

Babysitter - If you worked with coding agents, you probably experienced this pain: the lack of a structured process control and non-deterministic workflows. Babysitter lets you define iterative workflows (research → spec → TDD loop → quality gate → deploy) that are deterministic, resumable across sessions, and auditable, which is critical for moving AI-assisted development from ad-hoc experimentation toward reliable, production-grade engineering workflows and complex features.

https://github.com/a5c-ai/babysitter

Making Money in Uncertain Times

Tue, 10 Feb 2026 08:08:32 GMT

With the growing adoption of AI and recent moves like Anthropic’s Cowork plugin marketplace, there’s a popular narrative that traditional SaaS is dead. The implication is that the combination of AI agents + marketplaces will commoditize software entirely, and the old subscription paradigms won’t survive.

I tend to believe that’s an overstatement. SaaS isn’t dying - it’s evolving. What is under threat is SaaS as we’ve known it: long-term seat licenses, one-size-fits-all tiers, and feature-driven pricing. AI makes core functionality easier to replicate and access, pushing raw features toward commodity status. The value and pricing increasingly lie in outcomes, workflows, integrations, and customer trust. More than in the past, pricing is a strategic tool. See a similar conversion in David Ondrej's post on Twitter (link in the first comment)

That’s why understanding the possible pricing models and their tradeoffs matters more than ever. I’ve been listening to Ulrik Lehrskov-Schmidt’s webinar series on Agentic AI pricing -“Making Money in Uncertain Times.” In a world where both cost structures (compute, labor, infrastructure) and capabilities (model performance, automation) are shifting rapidly, we can’t anchor pricing to static assumptions. Instead, pricing needs to reflect real value delivery, signal predictable economics to customers, and align engineering decisions with business outcomes.

This isn’t just a GTM or sales conversation - it’s also a core product and engineering conversation. How we design systems, how we think about metrics and outcomes, and how we package those outcomes for customers all influence pricing levers.

The State of Coding Agents Using Local LLMs — February 2026

Sun, 01 Feb 2026 14:41:46 GMT

Last update: February 1st, 2026

Coding agents are no longer a novelty - they’re everywhere. Over the past year, we’ve seen massive adoption across startups and enterprises, alongside real improvements in autonomy, reasoning depth, and multi-step code execution. Tools like Claude Code, Codex, Copilot, and Kiro are shipping updates at a relentless pace, and teams are increasingly comfortable letting agents refactor modules, write tests, and manage pull requests.

But there’s a catch: these tools are token eaters. Autonomous agents don’t just answer a prompt - they plan, reflect, re-read the codebase, call tools, retry, and iterate. At scale, that translates into serious API bills.

That’s why we’re seeing growing interest in a different deployment pattern: running coding agents against local or self-hosted models. Ollama recently announced ollama launch a command that sets up and runs coding tools such as Claude Code, OpenCode, and Codex with local or cloud models. vLLM, LiteLLM, and OpenRouter also provide similar integrations. That signals that this is no longer fringe experimentation. For many teams, local LLMs are emerging as a viable path to reduce cost, improve stability, and gain tighter control over privacy.

Deployment models for coding agents

When teams talk about “running models locally,” they often mean different things. In practice, there are three distinct deployment patterns - and they differ meaningfully in cost structure, performance profile, and governance posture.

Local (Developer Machine) - the model runs directly on a developer’s laptop or workstation (e.g., via Ollama).
Hosted (Org-Managed Infrastructure / VPC) - the organization runs the model on its own infrastructure, either on-premises GPU servers or in a private cloud/VPC (e.g., via vLLM, Kubernetes, or managed GPU clusters).
Managed LLM API (e.g., Anthropic, OpenAI, etc.) - the model runs fully managed by a provider; the organization interacts via API.

Dimension	Local (Dev Machine)	Hosted (Org VPC / On-Prem)	Managed LLM API
Cost Structure	No per-token fees. Hardware cost borne by the developer. Cheap at a small scale; uneven across the team.	No per-token fees. Significant infra + ops cost. Economical at scale if usage is high.	Usage-based (per token / per request). Predictable but can become very expensive with agent loops.
Cost at Scale (Agents)	Hard to standardize; limited by laptop GPU/CPU.	Strong cost efficiency at high volume	Token costs compound quickly. Expensive in large org rollouts.
Performance (Latency)	Very low latency locally, but limited by hardware. Large models may be slow or impossible.	Good latency if well-provisioned GPU cluster. Can optimize with batching.	Typically excellent latency and throughput; globally distributed infra.
Model Size / Capability	Limited to smaller models (7B–34B typically; maybe 70B with strong GPUs).	Can run large open models (70B+), depending on infra budget.	Access to frontier SOTA models (often strongest reasoning & coding quality).
Quality (Coding Tasks)	Improving. “Good enough” for many workflows, especially with fine-tuned coding models.	Strong - can choose best open models and fine-tune internally.	Often highest raw reasoning quality and reliability on complex multi-file tasks.
Security / Privacy	Code never leaves device. Strong for IP protection. Risk: inconsistent security posture across developers.	Code stays inside org boundary. Strong centralized control.	Code leaves org boundary (even with enterprise contracts). Vendor trust required.
Compliance (GDPR, HIPAA, etc.)	Hard to audit across distributed machines.	Strong compliance posture if infra is controlled and logged centrally.	Enterprise compliance available via contract, but still external processing.
Governance & Observability	Weak - hard to monitor usage or enforce policies.	Strong - full logging, auditing, access controls, IAM integration.	Strong observability dashboards from vendor, but limited transparency into internals.
Stability / Availability	Works offline. Dependent on developer hardware reliability.	Controlled SLAs internally. Requires DevOps maturity.	Vendor-managed SLAs. Risk of outages outside your control.
Standardization Across Team	Low: “works on my machine” problem possible.	High - central model versions and infra.	Very high - single API endpoint for entire org.

Tools overview

Coding Agents and Model support

Coding Agent	Local LLM Support	Hosted Support	Notes
Claude Code	✅ via Ollama/vLLM integration	Native Anthropic	Run Claude Code with Local LLMs Using Ollama LLM gateway configuration LiteLLM Claude Code Quickstart OpenRouter integration with Claude Code
GitHub Copilot (Agent mode)	✅ via Ollama/vLLM integration	Cloud models (GPT-4o, Claude 3.5, Gemini, etc)	Ollama in VSCode GitHub copilot with Open Router GitHub copilot LLM Gateway
Codex (OpenAI)	✅ via Ollama integration	Cloud via OpenAI	Ollama Codex integration
Cursor AI	✅ via Ollama integration	Cloud multi-model	Use Local LLM with Cursor and Ollama OpenRouter with Cursor
AWS Kiro	❌ local	AWS hosted

Local LLM Frameworks

Framework	Primary Role	Notes
Ollama	Local LLM hosting & runtime	Lightweight CLI + API that serves models locally; integrates with multiple agents (Claude Code, Codex, Droid, OpenCode) and supports on-prem inferencing with moderate hardware.
vLLM (Serving)	High-performance LLM server	Optimized for scalable reasoning and long context LLM inference; integrates with agents (e.g., Claude Code) via Anthropic-Messages API compatibility.
OpenRouter	Unified LLM API broker	Central API layer for 400+ LLMs including local and cloud endpoints; can route agents to preferred backends with cost/redundancy optimization.
LiteLLM	Unified LLM API	Enables developers to use many LLM APIs, such as OpenAI, Anthropic, Gemini, and Ollama, in a single, OpenAI-compatible format.

Notable models

Model	Primary Use	Latest Release
Qwen3-Coder	Alibaba's 480B-parameter MoE coding model. SOTA results among open models on agentic coding tasks	July 2025
DeepSeek Coder	DeepSeek's open-source code model series (1B–33B params), achieving top performance among open-source code models across major benchmarks.	June 2024
Code Llama (7B/34B)	Meta's open-source code-specialized LLMs, fine-tuned from Llama 2 in multiple sizes	January 2024
gpt-oss	OpenAI's open-weight LLMs, available in 20B and 120B sizes under Apache 2.0. 120B variant matching o4-mini on reasoning benchmarks	August 2025
kimi-k2.5	Moonshot AI's open-source, native multimodal agentic model	January 2026

📈 Predictions Through 2026

1. Hybrid Routing Will Become the Standard

Cost is the most immediate driver. Autonomous coding agents are token-intensive by design. At enterprise scale, those token costs compound quickly.

Local inference eliminates per-token fees, which makes it attractive for high-volume, repetitive tasks. But frontier proprietary models still maintain an edge on complex, cross-repository reasoning and edge cases. The likely outcome is not full replacement, but intelligent routing:

Simpler or repetitive tasks → local or hosted open models
High-stakes, complex reasoning → managed frontier APIs

Tools like OpenRouter and LiteLLM are already enabling this pattern, and by the end of 2026, hybrid routing is likely to be the default deployment strategy for medium- to large-sized engineering organizations.

2. Standardization Will Lower the Switching Cost

Hybrid only works if switching models is frictionless.

As coding agents like Claude Code, Codex, Copilot, and others converge around shared inference interfaces (Ollama, vLLM, OpenAI-compatible endpoints), swapping models in and out becomes operationally simple. This reduces lock-in and makes experimentation safer.
As interoperability improves, the barrier to trying local models drops dramatically - and adoption follows.

3. Open-Source Coding Models Will Close the Gap

Tool-use fine-tuning is maturing. Code reasoning benchmarks are becoming more rigorous.

By late 2026, open-weight coding models are likely to be “production-grade” for a substantial share of workflows - especially where cost control and data sovereignty matter more than absolute frontier performance.

4. Resilience Will Matter as Much as Cost

There’s also a structural pressure building: agent-driven workloads amplify the impact of API outages. When a coding agent is embedded into CI pipelines or developer workflows, downtime is no longer an inconvenience - it’s a blocker.

As usage scales, reliance on a single managed API becomes a risk vector. This will accelerate investment in redundancy:

Secondary API providers
Local fallback models
On-prem capacity for critical workflows

Summary

In 2026, hybrid won’t just be about cost optimization - it will be about operational resilience.

The future is not “local vs cloud.” It’s a composable, policy-driven model infrastructure.

Organizations that treat model routing, hosting strategy, and redundancy as part of their core engineering architecture - rather than as an afterthought - will have structural advantages in cost control, privacy, and reliability.

2026 won’t be the year enterprises abandon managed APIs. It will be the year they stop depending on them exclusively.

Learn In Public - week 04

Tue, 27 Jan 2026 21:06:38 GMT

A sense of humor is one of the most underestimated leadership skills

Following the recommendation in “The Great CEO Within” I listened to “The One Minute Manager”. In one of the chapters, they mention the usage of Humor as a leadership tool. It is not about becoming a comedian but rather showing up as humans. Humor is a powerful and often overlooked tool.
Why it matters:

Humor helps build trust and rapport - people are more likely to engage and collaborate when they feel comfortable and connected.
It can reduce stress and tension, boosting well-being and performance.
Humor makes leaders more approachable and memorable, signaling confidence and emotional intelligence.
Shared laughter fosters psychological safety, helping teams voice ideas and take risks.

Of course, balance is key - humor should complement clarity and respect, not replace them. Too many jokes or poorly timed humor can actually backfire, so think of it as a strategic leadership tool, not a default setting. It’s about knowing when a light moment can lower defenses, reset the room, or simply remind everyone that work is done by humans, not robots.

Related resources -

Laughter in Leadership & Workplace Performance - https://www.forbes.com/sites/cherylrobinson/2025/03/09/laughter-in-leadership-how-humor-enhances-workplace-performance
How Humor Keeps Employees Engaged - https://www.shrm.org/mena/topics-tools/news/employee-relations/how-humor-can-keep-employees-engaged
Humor Boosts Credibility at Work - https://www.atlassian.com/blog/teamwork/humor-at-work

Tokens as Currency

Mon, 26 Jan 2026 08:32:08 GMT

Half-baked thought: Tokens will become currency.

Right now, the direction is obvious - more money buys more tokens.

But what if, in the near future, tokens themselves become a medium of exchange?

Consider this:

Microsoft is allocating tokens to support the maintenance of open-source projects.
Companies granting tokens in exchange for using their tools or infrastructure.
Open-source maintainers receiving donations in tokens instead of (or alongside) cash.
Platforms enabling distributed token usage across multiple accounts, almost like a modern SETI@home
Gift cards for Anthropic.

In other words, tokens are not just consumption units, but are tradable, transferable assets within an ecosystem.

Of course, this is far from trivial. Privacy, security, incentive alignment, and implementation complexity are all major hurdles.

But if I had to place one slightly outrageous bet for 2026, it would be movement in this direction.

Are we ready to start thinking of tokens as something you can “round up and donate”?

Learn in Public – week 03

Tue, 20 Jan 2026 15:27:36 GMT

I finished listening to “The Great CEO Within” by Matt Mochary and Alex MacCaw. A few thoughts:

1️⃣ A tactical cheat sheet
I view the book as a tactical cheat sheet: short, practical chapters you can skim for ideas. It’s great for quick exposure, and most chapters include references for deeper dives. For me, this book has excellent value for time.

2️⃣ Revisit in the LLM era
Two well-known ideas in the book are Getting Things Done and Inbox Zero.
Inbox-zero and productivity advice hit differently today. With LLMs helping triage emails, summarize threads, and highlight what actually matters, the principles remain the same - but the execution is far more automated.

3️⃣ Optimizing meetings
TL;DR: come prepared, use written communication in advance, and don’t deviate from the planned agenda.
The authors suggest holding all meetings, including 1:1s, on the same day. From my experience, for 1:1s that go beyond status updates and require real attention (e.g., feedback), stacking too many of them on the same day can be overwhelming for most people.

4️⃣ “The first thing to optimize is yourself”
One of my favorite quotes from the book. It emphasizes founders’ and leaders’ health, mental and physical, something that has historically been overlooked. A good reminder that sustainable leadership starts with managing your own energy.

The book also mentions principles of conscious leadership: listening to feedback and acting on it, and not being afraid to make (and admit) mistakes.
This week, I also read a blog titled “Reflection is a Crucial Leadership Skill”, which made these ideas more actionable and down-to-earth.

What should I read next?

Learn in Public - week 02

Wed, 14 Jan 2026 10:34:54 GMT

I started this week with deeplearning.ai’s course on semantic caching, created in collaboration with Redis. That sent me down a rabbit hole, exploring different LLM caching strategies and the products that support them.

One such product is AWS Bedrock Prompt Caching. If large parts of your prompts are static (specifically, the prefixes), retokenizing the prefix on every request is a waste of time and money. Prompt or context caching lets you process the prefix once and store it, reducing costs and improving performance.

Sounds great, right? Let's check the pricing mode. If your requests are more than 5 minutes apart, your cache will be cleared. If your requests are short, caching won’t be activated; if the cache hit rate is low, you will pay an extra, non-usage-based premium for cache writes. I highly recommend reading the “How Much Does Bedrock Prompt Caching Cost?” section in the article “Amazon Bedrock Prompt Caching”.

AI Just Took a Big Step Into Digital Health

Tue, 13 Jan 2026 09:16:04 GMT

🚀 AI is moving deeper into digital health - over the past week, both OpenAI and Anthropic have introduced major features aimed at bringing powerful AI capabilities into healthcare and life sciences. Links in the first comment

🔹 OpenAI: ChatGPT Health
OpenAI has launched ChatGPT Health, a dedicated health experience that lets users securely connect their medical records and wellness app data (e.g., Apple Health, Function, and MyFitnessPal) to get more informed insights about their health and wellness. The feature is designed to help people better interpret test results, prepare for doctor visits, and navigate everyday health questions — not replace clinicians. Enhanced privacy protections ensure that health chats and data remain isolated and encrypted, and that users retain full control over connections and data.

🔹 Anthropic: Claude for Healthcare & Life Sciences
Following an earlier announcement regarding Claude for Life Sciences, Anthropic introduced Claude for Healthcare alongside expanded life science capabilities, bringing its Claude AI into regulated medical and scientific use cases. This includes HIPAA-ready infrastructure and connectors to industry data sources (like CMS coverage rules, ICD-10 codes, and NPI registries) to support tasks such as prior authorizations, claims management, and clinical documentation. Claude can also summarize medical histories and explain test results in plain language. On the life sciences side, new integrations with clinical trial, preprint, and bioinformatics platforms aim to accelerate research workflows and regulatory documentation.

Both announcements show the AI industry racing into digital health with different focus areas. OpenAI’s move toward personalized health guidance for individuals complements Anthropic’s broader, enterprise-oriented tools for providers and researchers. Together, they raise exciting possibilities and important questions about regulatory standards, data privacy, and the role of AI in care delivery.

Bonus - GrantFlow - a grant management platform that automates discovery, planning, and application workflows for researchers and institutions.

4 AWS re:Invent announcment to check

Fri, 05 Dec 2025 21:20:25 GMT

AWS re:Invent 2025 took place this week, and as always, dozens of announcements were unveiled. At the macro level, announcing Amazon EC2 Trn3 UltraServers for faster, lower-cost generative AI training can make a significant difference in the market, which is primarily biased towards Nvidia GPUs. At the micro-level, I chose four announcements that I find compelling and relevant for my day-to-day.

AWS Transform custom - AWS Transform enables organizations to automate the modernization of codebases at enterprise scale, including legacy frameworks, outdated runtimes, infrastructure-as-code, and even company-specific code patterns. The custom agent applies those transformation rules defined in documentation, natural language descriptions, or code samples consistently across the organization's repositories.

Technical debt tends to accumulate quietly, damaging developer productivity and satisfaction. Transform custom wishes to “crush tech debt” and free up developers to focus on innovation instead. For organizations managing many microservices, legacy modules, or long-standing systems, this could dramatically reduce the maintenance burden and risk and increase employees’ satisfaction and retention over time.

https://aws.amazon.com/blogs/aws/introducing-aws-transform-custom-crush-tech-debt-with-ai-powered-code-modernization

Partially complementary, AWS introduced 2 frontier agents in addition to the already existing Kiro agent -

DevOps agent - an on-call / incident-response agent that integrates across monitoring, code repos, and service tickets to detect root causes and coordinate responses.
https://aws.amazon.com/blogs/aws/aws-devops-agent-helps-you-accelerate-incident-response-and-improve-system-reliability-preview/
Security agent - an agent that proactively secures applications throughout the entire development lifecycle by conducting automated design and code reviews, and performing on-demand penetration testing
https://aws.amazon.com/blogs/aws/new-aws-security-agent-secures-applications-proactively-from-design-to-deployment-preview/

AWS Lambda Durable Functions - Durable Functions enable building long-running, stateful, multi-step applications and workflows - directly within the serverless paradigm. Durable functions support a checkpoint-and-replay model: your code can pause (e.g., wait for external events or timeouts) and resume within 1 year without incurring idle compute costs during the pause.

Many real-world use cases, such as approval flows, background jobs, human-in-the-loop automation, and cross-service orchestration, require durable state, retries, and waiting. Previously, these often required dedicated infrastructure or complex orchestration logic. Durable Functions enable teams to build more robust and scalable workflows and reduce overhead.

https://aws.amazon.com/blogs/aws/build-multi-step-applications-and-ai-workflows-with-aws-lambda-durable-functions

AWS S3 Vectors (General Availability) - Amazon S3 Vectors was announced about 6 months ago and is now generally available. This adds native vector storage and querying capabilities to S3 buckets. That is, you can store embedding/vector data at scale, build vector indexes, and run similarity search via S3, without needing a separate vector database. The vectors can be enriched with metadata and integrated with other AWS services for retrieval-augmented generation (RAG) workflows. I think of it as “Athena” for embeddings.

This makes it much easier and cost-effective for teams to integrate AI/ML features - even if they don’t want to manage a dedicated vector DB and reduces the barrier to building AI-ready data backends.

https://aws.amazon.com/blogs/aws/amazon-s3-vectors-now-generally-available-with-increased-scale-and-performance

Amazon SageMaker Serverless Customization - Fine-Tuning Models Without Infrastructure - AWS announced a new capability that accelerates model fine-tuning by eliminating the need for infrastructure management. Teams can upload a dataset and select a base model, and SageMaker handles the fine-tuning pipeline, scaling, and optimization automatically - all in a serverless, pay-per-use model. This customized model can also be deployed using Bedrock for Serverless inference. It is a game-changer, as serving a customized model was previously very expensive. This feature makes fine-tuning accessible to far more teams, especially those without dedicated ML engineers.

https://aws.amazon.com/blogs/aws/new-serverless-customization-in-amazon-sagemaker-ai-accelerates-model-fine-tuning

These are just a handful of the (many) announcements from re:Invent 2025, and they represent a small, opinionated slice of what AWS showcased. Collectively, they highlight a clear trend: Amazon is pushing hard into AI-driven infrastructure and developer automation - while challenging multiple categories of startups in the process.

While Trn3 UltraServers aim to chip away at NVIDIA’s dominance in AI training, the more immediate impact may come from the developer- and workflow-focused releases. Tools like Transform Custom, the new frontier agents, and Durable Functions promise to reduce engineering pain - if they can handle the real, messy complexity of enterprise systems. S3 Vectors and SageMaker Serverless Customization make it far easier to adopt vector search and fine-tuning without adding a new operational burden.

5 interesting things (02/11/2025)

Sun, 02 Nov 2025 14:20:59 GMT

Measuring Engineering Productivity - Measuring engineering productivity is a topic that has been discussed as long as the field of engineering has existed. This post acknowledges the tension of measuring engineering work (where metrics can be easily manipulated). It proposes a pragmatic system of minimal burden, high visibility, and context-sensitive metrics, rather than focusing on lines of code.

https://justoffbyone.com/posts/measuring-engineering-productivity/

Stop Avoiding Politics - Politics usually have a bad name, but the article argues that avoiding the “politics” of an organization doesn’t remove politics; it just removes your ability to influence outcomes and let others decide for you. This is a helpful reminder that part of seniority is engaging in stakeholder dynamics, not just writing code.

https://terriblesoftware.org/2025/10/01/stop-avoiding-politics/

Team Dynamics after AI - This post critiques the rush to utilize AI to scale engineering artifact production and argues that what matters most remains the “illegible” human and team elements: context, feedback loops, diversity of roles, and the social glue that holds work together. I link it to the "Measuring engineering productivity" post in the sense that simply measuring throughput or artifacts might miss the hidden “team health” or context dimension.

https://mechanicalsurvival.com/blog/team-dynamics-after-ai/

Useful Engineering Management Artifacts - This is a practical collection of templates for various purposes, including team charters, career development plans, and decision briefs. It complements the productivity and team dynamics posts by providing actual artifacts you can use to operationalize some of the ideas.

https://bjorg.bjornroche.com/management/engineering-management-artifacts/

Stop Caring So Much About Your People - I find this post a bit weird. In the post "Radical Candor" era, it feels obvious that giving feedback is both essential and meaningful. I agree with the author’s point that leaders sometimes over-prioritize team happiness at the expense of organizational health, and I’d extend that further: over-protecting people from discomfort also hurts their own growth. As leaders, giving feedback is often uncomfortable, but it’s one of the most valuable things we can do - to help our people, our teams, our company, and even ourselves evolve.

https://avivbenyosef.com/stop-caring-so-much-about-your-people/

From Demo Hell to Scale: Two Takes on Building Things That Last

Tue, 16 Sep 2025 11:03:49 GMT

I recently came across two blog posts that made me think, especially in light of a sobering statistic I've seen floating around: a recent MIT study reports that 95% of enterprise generative AI pilots fail to deliver real business impact or move beyond demo mode.

One post is a conversation with Werner Vogels, Amazon’s long-time CTO, who shares lessons from decades of building and operating systems at internet scale. The other, from Docker, outlines nine rules for making AI proof-of-concepts that don’t die in demo land.

Despite their different starting points, I was surprised by how much the posts resonated with one another. Here’s a short review of where they align and where they differ.

Where They Agree

Solve real problems, not hype - Both warn against chasing the “cool demo.” Docker calls it “Solve Pain, Not Impress”, while Vogels is blunt: “Don’t build for hype.” This advice sounds obvious, but it’s easy to fall into the trap of chasing novelty. Whether you’re pitching to executives or building at AWS scale, both warn that if you’re not anchored in a real customer pain, the project is already off track.
Build with the end in mind - Neither believes in disposable prototypes. Docker advises to design for production from day zero—add observability, guardrails, testing, and think about scale early. Vogels echoes with “What you build, you run”, highlighting that engineers must take ownership of operations, security, and long-term maintainability. Both perspectives converge on the same principle: if you don’t build like it’s going to live in production, it probably never will.
Discipline over speed - Both posts emphasize discipline over blind speed. Docker urges teams to embed cost and risk awareness into PoCs, even tracking unit economics from day one. Vogels stresses that “cost isn’t boring—it’s survival” and frames decision-making around reversibility: move fast when you can reverse course, slow down when you can’t. Different wording, same idea: thoughtful choices early save pain later.

Where They Differ

Scope: the lab vs. the long haul - Docker’s post is tightly focused on how to build POCs in the messy realities of AI prototyping and how to avoid “demo theater” and make something that survives first contact with production. Vogels’ advice is broader, aimed at general engineering, technology leadership, infrastructure, decision-making at scale, and organization-level priorities. Vogels speaks from decades of running Amazon-scale systems, where the horizon is years, not weeks.
Tactics vs. culture - Docker’s advice is concrete and technical: use remocal workflows, benchmark early, add prompt testing to CI/CD. Vogels is less about specific tools and more about culture: engineers owning what they build, organizations learning to move fast on reversible decisions, and leaders setting clarity as a cultural value. Docker tells you what to do. Vogels tells you how to think.
Organizational Context and Scale - Docker speaks to teams fighting to get from zero to one—making PoCs credible beyond the demo stage. Vogels speaks from AWS’s point of view, where the challenge is running infrastructure that millions rely on. Docker’s post is about survival; Vogels is about resilience at scale.

What strikes me about these two perspectives is how perfectly they complement each other. Docker's advice isn't really about AI - it's about escaping demo hell by building prototypes with production DNA from day one. Vogels tackles what happens when you actually succeed: keeping systems reliable when thousands depend on them. They're describing the same journey from different ends. Set up your prototypes with the right foundations, and you dramatically increase the odds that your product will one day face the kinds of scale and resilience questions Vogels addresses.

AI, Paradigm Shifts, and the Future of Building Companies

Mon, 25 Aug 2025 14:09:52 GMT

Over the past few months, I have been constantly reading conversations about how Generative AI will reshape software engineering. On LinkedIn, Twitter, or in closed professional groups, engineers and product leaders debate how tools like Cursor, GitHub Copilot, or automated testing frameworks will impact the way software is built and teams are organized.

But the conversation goes beyond just engineering practices. If we zoom out, AI will not only transform the workflows of software teams but also the structure of companies and even the financial models on which they are built. This kind of change feels familiar - it echoes a deeper historical pattern in how science and technology evolve.

Kuhn’s Cycle of Scientific Revolutions

During my bachelor’s, I read Thomas Kuhn’s The Structure of Scientific Revolutions. Kuhn argued that science does not progress in a linear, step-by-step manner. Instead, it moves through cycles of stability and disruption. The Kuhn Cycle¹, as reframed by later scholars, breaks this process into several stages:

Pre-science - A field without consensus; multiple competing ideas.
Normal Science - A dominant paradigm sets the rules of the game, guiding how problems are solved.
Model Drift - Anomalies accumulate, and cracks in the model appear.
Model Crisis - The old framework fails; confidence collapses.
Model Revolution - New models emerge, challenging the old order.
Paradigm Change - A new model wins acceptance and becomes the new normal.

The Kuhn Cycle Applied to Software Development

Normal Science

For decades, software engineering has operated under a shared set of practices and beliefs:

Clean Code & Best Practices - DRY, SOLID, Unit Testing, Peer Reviews.
Agile & Scrum - Iterative sprints and ceremonies as the “right” way to build products.
DevOps & CI/CD - Automation of builds, deployments, and testing.
Organizational Structure – Specialized roles (frontend, backend, QA, DevOps, PM) and a belief that more engineers equals more output.

The underlying assumption is hire more engineers + refine practices → better and quicker software.

Model Drift

Over time, cracks began to show.

The talent gap - demand for software far outstrips available developers.
Velocity mismatch - Agile rituals can’t keep pace with market demands.
Complexity overload - Microservices and massive codebases create systems that are too complex for a single person to comprehend fully.
Knowledge silos - onboarding takes months, and institutional knowledge remains fragile.

These anomalies signaled that “hire more engineers and improve processes” was no longer a sustainable model.

Model Crisis

The strain became obvious:

Even tech giants with thousands of engineers struggle with code sprawl and coordination overhead.
Brooks’ Law bites - adding more people to a project often makes it slower.
Business pressure grows - leaders demand faster iteration, lower costs, and higher adaptability than human-only teams can deliver.
Early AI tools, such as GitHub Copilot and ChatGPT, reveal something provocative - machines can generate boilerplate, tests, and documentation in seconds - tasks once thought to be unavoidably human.

This is where many organizations sit today - patching the old paradigm with AI, but without a coherent new model.

Model Revolution

A new way of working begins to take shape. Here are some already visible in experimenting, we can all see around us -

AI-first engineering - using AI agents for scaffolding code, generating tests, or refactoring large systems. Humans act as curators, reviewers, and high-level designers.
Smaller, AI-augmented teams
New roles and workflows - QA shifts toward system-level validation; PMs focus less on ticket grooming and more on problem framing and prompting.
Org structures evolve - less siloing by specialization, more “AI-augmented full-stack builders.”
Economics shift - productivity is no longer headcount-driven but iteration-driven. Cost models change when iteration is nearly free.

Paradigm Change

In the coming years, some of the ideas above, and probably additional ideals, could stabilize as the “normal science” of software development and organizational building. But we are not yet there. Once we get there, today’s experiments will feel as obvious as Agile sprints or pull requests do now.

We are in the midst of model drift tipping into crisis, with glimpses of revolution already underway. Kuhn’s lesson is that revolutions are not just about better tools - they’re about shifts in worldview. For AI, the shift might be that companies will no longer be limited by headcount and manual processes but by their ability to ask the right questions, frame the correct problems, and adapt their models of value creation.

We are moving toward a future where the shape of companies, not just their software stacks, will look radically different, and that’s an exciting era to be a part of.

LLM Debt: The Double-Edged Sword of AI Integration

Sun, 17 Aug 2025 11:11:21 GMT

Have you noticed how half the posts on LinkedIn these days feel like they were written by an LLM - too many words for too little substance? Or how product roadmaps suddenly include “AI features” that nobody asked for, just because it sounds good in a pitch deck? Or those meetings where someone suggests “let’s use GPT for this,” when a simple SQL query, an if-statement, or a much simpler ML model would do the job?

Laurence Tratt recently coined the term LLM inflation to describe how humans use LLMs to expand simple ideas into verbose prose, only for others to shrink them back down. That concept got me thinking about a related phenomenon: LLM debt.

LLM debt is the growing cost of misusing LLMs — by adding them where they don’t belong and neglecting them where they could help.

We’re all familiar with technical debt, product debt, and design debt¹ — the shortcuts or missed opportunities that slow us down over time. Similarly, organizations are quietly accumulating LLM debt.

So, what does LLM debt look like in practice? It’s a double-edged liability:

Overuse: Integrating LLMs where they’re unnecessary adds latency, complexity, cost, and stochasticity to systems that could be simpler, faster, and more reliable without them. For example, sending every API request through a multimillion-parameter model when a simple regex or deterministic logic would suffice.
Underuse: Failing to adopt LLM-based tools where they could genuinely help results in wasted effort and missed opportunities. Think of teams manually triaging support tickets, writing repetitive documentation, or analyzing text data by hand when an LLM could automate much of the work.

Like product or technical debt, a small amount of LLM debt can be strategic: it allows experimentation, faster prototyping, or proof-of-concept development. However, left unmanaged, it compounds, creating systems that are over-engineered in some areas and under-leveraged in others, which slows product evolution and innovation. Same as other types of debt, it should be owned and managed.

LLMs are powerful, but they come with costs. Just as we track and manage technical debt, we need to recognize, measure, and pay down our LLM debt. That means asking tough questions before adding LLMs to the stack, and also being bold enough to leverage them where they could provide real value.

If LLM inflation showed us how words can expand and collapse in unhelpful cycles, LLM debt shows us how our systems can quietly accumulate inefficiencies that slow us down. Recognizing it early is the key to keeping our products lean, intelligent, and future-ready.

Things I recently built with my daughter

Mon, 04 Aug 2025 14:04:31 GMT

My daughter just finished first grade and is about to turn seven. One day, she noticed me playing a math puzzle on a news site and asked if she could join. I told her that when an easy one came along, I’d be happy to solve it with her.

A few days passed. She asked again.

I didn’t want to say no, so I offered to sketch a similar game on paper. But designing a good puzzle turned out to be more challenging than expected. I scribbled a lightweight version instead—and that sparked an idea. I asked if she’d like to build a game together.

1. Math Grid

🔗 Play Math Grid

This was our first game. It’s a visual puzzle: players are given a grid with some numeric constraints on rows and columns, and the goal is to fill it in correctly. Inspired by logic puzzles like Sudoku and Kakuro, the constraints are simplified for early elementary-level arithmetic, so it stays approachable while still encouraging reasoning.

2. Number Detector

🔗 Play Number Detector

Later, while solving exercises from her school workbook, we came across number riddles. This inspired our second app. In Number Detector, the player is given partial clues (like a number’s sum of digits or a multiple constraint) and has to figure out the correct number from a limited set.

We added randomness and regeneration to keep things fresh, making the practice repeatable and varied.

3. Triangle Tactics

🔗 Play Triangle Tactics

A month later, while playing a pen-and-paper strategy game, she casually said: “Let’s make it into an app—like we did in the old days.” :)

Her request blew my mind.

We built Triangle Tactics, a simple turn-based game where players connect dots to form triangles, and the player who forms the most triangles wins. What surprised me most was the technical challenge—not in the game logic, but in rendering the dots and lines correctly. It was solved only after switching to a different LLM model than the default.

Reflections

These experiments turned into more than just games. They became opportunities to:

Understand the importance of English: We worked in English—despite it not being our native language—and talked about why fluency matters, especially when working with tools, prompts, or documentation.
Explore computers as collaborators: “What do you mean it misunderstood the prompt?”, “How should we ask to get the result we want?”
Practice QA thinking: “Why isn’t this button working?”, “Is it behaving the way we expected?”
Enjoy quality time: A fun excuse to step outside our usual routines and build something together.

AWS has entered the building

Wed, 16 Jul 2025 20:01:00 GMT

AWS has released several notable announcements within the LLM ecosystem over the last few days.

Introducing Amazon S3 Vectors (preview) - Amazon S3 Vectors is a durable, cost-efficient vector storage solution that natively supports large-scale AI-ready data with subsecond query performance, reducing storage and query costs by up to 90%.

Why I find it interesting -

Balancing cost and performance - i.e., storing on a database is more expensive but yields better results. If you know what the "hot vectors" are, you can store them in the database and store the rest in S3.
Designated buckets - it started with table buckets and has now evolved to vector buckets. Interesting direction.

Launch of Kiro - the IDE market is on fire with OpenAI’s acquisition falling apart, Claude code and cursor competition, and now Amazon reveals Kiro with the promise - "helps you do your best work by bringing structure to AI coding with spec-driven development"

Why I find it interesting -

At first, I wondered why AWS entered this field, but I assume it is a must-have these days, and might lead to higher adoption of their models or Amazon Q.
The different IDEs and CLI tools are influenced by each other so it will be interesting to see how a new player influences this space.

Strand agents are now at v1.0.0 - Strand Agents are an AWS open-source SDK that enables building and running AI agents across multiple environments and models, with many pre-built tools that are easy to use.

Why I find it interesting -

The bedrock agents interface was limiting for a production-grade agent, specifically in terms of deployment modes, model support, and observability. Strand agents open many more doors.
There are many agent frameworks out there (probably two more were released while you read this post). Many of them experience different issues when working with AWS Bedrock. If you are using AWS as your primary cloud provider, it should be a leading candidate.

5 interesting things (11/05/2025)

Sun, 11 May 2025 09:34:24 GMT

Agents app design pattern - this is a back-to-basics adaptation. How would we read this 14 years from now? Would the ideas he mentioned there be a standard?

https://github.com/humanlayer/12-factor-agents

The original document "12 factors app" is also worth reading (note that it was first published 2011+-) -

https://12factor.net

When the Agents Go Marching In: Five Design Paradigms Reshaping Our Digital Future

This post complements the previous one, covering the same topics. If you are in a hurry, jump to the "The Reinvention of UX: Five Emerging Paradigms" section. I feel that I cope with all those aspects, e.g., building trust, transparency, cognitive load distribution, etc., on a daily basis.

https://medium.com/cyberark-engineering/when-the-agents-go-marching-in-five-design-paradigms-reshaping-our-digital-future-a219009db198

Using Alembic to create and seed a database
Seeding a database is essential for testing, development, and ensuring consistent application behavior across different environments. Alembic is a lightweight database migration tool for Python, designed to work seamlessly with SQLAlchemy.

We use Alembic to manage our database migrations, and I recently needed to seed our database for consistency across environments. I looked for several solutions and eventually used the solution in this post to create a migration that seeds the database -

https://medium.com/@fimd/using-alembic-to-create-and-seed-a-database-8f498638c406

A Field Guide to Rapidly Improving AI Products - while this post focuses on AI products, specifically ones LLM-based, multiple lessons can also be adapted to non-LLM-based AI products and general products. Conducting an error analysis, generating synthetic data (preferably with domain express), and using a data viewer are good starting points.

https://hamel.dev/blog/posts/field-guide/

I Tried Running an MCP Server on AWS Lambda… Here’s What Happened - this post involves two topics I think a lot about these days - MCP and serverless computing. I think it is clear why I think a lot about MCPs. But why do I think about serverless computing? I think of it as a low-cost solution for early-stage startups. Early-stage startups usually have low traffic, which does not justify the cost of having servers up 24/7. On the other hand, the serverless development experience still needs some refinement, and there are services that one would like to host that do not support running in a serverless manner.

https://www.ranthebuilder.cloud/post/mcp-server-on-aws-lambda

5 interesting thing (28/03/2025)

Fri, 28 Mar 2025 10:51:45 GMT

PgAI - LLMs have been part of everyday life already for a while. One aspect I think has not been explored well so far is using them as part of ETL. The implementations I have seen so far don't take advantage of batch APIs and are not standardized to enable the easy replacement of a model. Having said that, I believe those hurdles will be overcome soon.

https://github.com/timescale/pgai

Related links

Snowflake cortex complete - call llm functions from snowflake https://docs.snowflake.com/en/developer-guide/snowpark-ml/reference/1.5.0/api/model/snowflake.cortex.Complete)
OpenAI Postgres extension - allows calling OpenAI or local models using Ollama. https://www.crunchydata.com/blog/accessing-large-language-models-from-postgresql
Do you use LLMs in your ETL pipelines - Reddit discussion regarding using LLMs in ETL pipelines. There are pros and cons of course, specially for cases you need a deterministic answer but there are also other cases -
https://www.reddit.com/r/dataengineering/comments/1h17gjm/do_you_use_llms_in_your_etl_pipelines/

Life Altering PostgreSql Patterns - a back-to-basics post. I agree with most of the points mentioned there, specifically around adding creaetd_at, updated_at, and deleted_at attributes to all tables and saving state data as logs rather than saving only the latest state. I found the section about enum tables interesting. This is the first time I was exposed to this idea, and the ability to add a description or metadata is excellent.

https://mccue.dev/pages/3-11-25-life-altering-postgresql-patterns

Via this post, I learned about the on update cascade option, you can read more about it here - https://medium.com/geoblinktech/postgresql-foreign-keys-with-condition-on-update-cascade-330e1b25b6e5

AI interfaces of the future - I usually don't share videos, but I think this talk is thought-provoking for several reasons -

Gen UI patterns - an emerging field, the talk reviews several products and highlights good and destructive patterns. Some of the patterns, like suggestions or auto-complete, are transparent to us but are present in many products we know, and that's something important to notice when you build such a product.
Product review: Knowing what is out there is good for inspiration, ideas, and understanding the competitive landscape. However, new products are coming out every day, and it is hard to track all of them.

https://www.youtube.com/watch?v=DBhSfROq3wU

Simplify Your Tech Stack: Use PostgreSQL for Everything - Two widespread tensions, especially in startups, are build vs. buy conflicts and using specialized products or technologies (e.g., different databases) that are top of the breed but not many people can use and maintain vs. more common technology that more people can maintain but can have performance drawbacks or other limitations. Mainly working in startups, I usually prefer to use standard technology to run faster, knowing that the product, focus, and priorities often change. With that being said, I acknowledge that early adoption of new technologies can be life-changing for a startup, but figuring out what to bet on is hard.

https://medium.com/timescale/simplify-your-tech-stack-use-postgresql-for-everything-f77c96026595

CDK Monitoring Constructs - if you are using AWS CDK as your IAC tool, CDK monitoring constructs enable you to create cloudwatch alarms and dashboards almost out of the box. I wish they would release and add additional options at a faster pace.

https://pypi.org/project/cdk-monitoring-constructs/

Better Plotly Bar Chart

Thu, 21 Nov 2024 21:22:05 GMT

I'm reading "storytelling with data" by Cole Nussbaumer Knaflic. I plan to write my thoughts and insights from the book once I finish it. For now, I wanted to play with it a bit and create a better bar chart visualization that -

Highlight the category you find most important and assign a special color to it (i.e prominent_color in the code), while the remaining categories used the same color (i.e. latent_color)
Remove grids and make the background and paper colors the same to remove cognitive load.

The implementation is flexible, so if you feel like changing one of the settings(i.e., show the grid lines or center the title) you can pass it via keyword arguments when calling the function.

https://gist.github.com/tomron/dffae6ca7c679094894159b26b1d4d6f

📚 Book club Q2 2024 – 3 Reading recommendations

Tue, 09 Jul 2024 08:21:14 GMT

This quarter, I read 3 books that relate to France and the resistance during the 2nd world War -

Code Name Hélène by Ariel Lawhon. This thriller is based on Nancy Wake's story. It also reminded me of "Agent Sonya" by Ben Macintyre, which I read a while ago and enjoyed.
The Paris Architect by Charles Belfoure - is a touching story with lots of imagination but I was a bit disappointed it was not based on a real story.
The Dressmaker's Secret by Rosalie Ham - this book tells the story of Coco Chanel during WWII from the prism of her personal assistant or more accurately from the granddaughter of her personal assistant who unpacks the story. I didn't know about her relationships with the Nazis, and it was interesting to learn about them.

I also listened to "Setting the Table" by Danny Meyer, and it was very insightful, it even made me publish a LinkedIn post.

5 interesting things (31/05/2024)

Fri, 31 May 2024 08:07:57 GMT

How we built Text-to-SQL at Pinterest - Text-to-SQL and vice versa became one of the canonical examples of LLM, and every product needs one. The post described a very interesting work that can be implemented relatively easily. I relate the most to the closing paragraph, which emphasizes the gap between demos, tutorials, benchmarks, and real-world use cases. - "It would be helpful for applied researchers to produce more realistic benchmarks which include a larger amount of denormalized tables and treat table search as a core part of the problem."

https://medium.com/pinterest-engineering/how-we-built-text-to-sql-at-pinterest-30bad30dabff

(p.s I mentioned post in a recent LinkedIn post - LLMs in the enterprise - looking beyond the hype on what's possible today)

How an empty S3 bucket can make your AWS bill explode - this story completely blew my mind (and gladly not my account). I was happy to see that AWS is looking into this issue and wondered if in bigger accounts, such anomalies could get unnoticed.

https://medium.com/@maciej.pocwierz/how-an-empty-s3-bucket-can-make-your-aws-bill-explode-934a383cb8b1

The Design Philosophy of Great Tables - great_tables is a Python package for creating wonderful-looking tables. This post shares its visual design philosophy and is worth reading if you create tables even if you will not use this package.

https://posit-dev.github.io/great-tables/blog/design-philosophy/

1-measure-3-1 - a variation of the 1-3-1 problem-solving method for making proposals. I found it specifically effective for engineers as it is structured and focused.

https://www.annashipman.co.uk/jfdi/1-measure-3-1.html

On Making Mistakes — I love it when people combine experience or knowledge in one field or domain with another. For example, someone brings her experience as a soccer player to managing a team, or someone uses lessons he learned as a supermarket cashier to software architecture. This post discusses making mistakes and working through them and refers to several domains, including improv, chess, and F1 team management.

https://read.perspectiveship.com/p/on-making-mistakes

📚 Book club Q1 2024 - 3 Reading recommendations

Sun, 31 Mar 2024 10:18:57 GMT

This year, I track my book reading for the first time. I don't know if I'll keep doing it after this year or if I'll last until the end of the year, but for now, I'm all in. This helps me reflect on my reading and remember the books I enjoyed.

Fundable: Why Some Entrepreneurs Get Funded, And Others Do Not! by Sephi Shapira

This is the best value for my time in a long time. The book is filled with concrete and practical advice, which I immediately found myself using.

https://www.amazon.com/Fundable-Some-Entrepreneurs-Funded-Others/dp/B0B4FD6ZV3

Beartown by Fredrik Backman -

That's the perfect book for me—it has lots of sports and a human story, some gender tension, and two more books in this series (Us Against You and Winners).

https://www.amazon.com/Beartown-Novel-Fredrik-Backman-ebook/dp/B01KG5GQDS

STFU: The Power of Keeping Your Mouth Shut in an Endlessly Noisy World by Dan Lyons

For long parts of this book, it was a paradox to me - why write 200+ pages to say we should all shut up - just do it.

I listened to this book after also listening to Disrupted by Lyons. I like his style and narration. When listening to Disrupted, I thought that it probably damaged his employability, which turned out to be true, as he discussed in STFU.

The book can sometimes be extreme or refer to an extreme crowd (50 out of 50 in the Talkaholics test). However, I liked the book because it discusses many aspects of our lives - friends and family, work, etc.- and a few things I can immediately adapt. For example, I lowered my cell phone usage near my kids and hope to stick with it.

In one of the last chapters, he mentions Never Split the Difference by Chris Voss and Tahl Raz, which I also recently listened to. I listened to Never Split the Difference after a friend told me that she was starting to reread this book, and after listening to it, I completely understood why she wanted to reiterate it.

I read the book to improve my negotiation skills. I'm not sure there is an immediate effect, but as the two books pointed out - I talk less and actively listen more. So I try to pause before I answer, be succinct and hum, and let the other person talk.

https://www.amazon.com/STFU-Power-Keeping-Mouth-Talking-ebook/dp/B09X427GK8

5 interesting things (08/03/2024)

Fri, 08 Mar 2024 10:39:32 GMT

(Almost) Every infrastructure decision I endorse or regret after 4 years running infrastructure at a startup - in my current role as a CTO of an early-stage startup, I make many choices about tools, programming languages, architecture, vendors, etc. This retrospective view was fascinating not only for the tools themselves but also for the arguments.

https://cep.dev/posts/every-infrastructure-decision-i-endorse-or-regret-after-4-years-running-infrastructure-at-a-startup/

Everything You Can Do with Python’s textwrap Module - I have used Python for more than 10 years and never heard of textwrap model. Maybe you, too, haven't heard of it.

https://towardsdatascience.com/everything-you-can-do-with-pythons-textwrap-module-0d82c377a4c8

It was never about LLM performance - I couldn't agree more. The performance gaps between different LLMs are becoming neglectable. Now, it is about the experience you build using those models and the guardrails you put in to ensure the experience.

https://read.technically.dev/p/it-was-never-about-llm-performance

How to build an enterprise LLM application: Lessons from GitHub Copilot - the post ends with a summary of 3 key takeaways -

Identify a focused problem and thoughtfully discern an AI’s use cases.
Integrate experimentation and tight feedback loops into the design process
As you scale, continue to leverage user feedback and prioritize user needs

Those takeaways are general and correct for almost every product launch I can think of. The post provides more concrete tips for LLM applications. It is interesting to read about a product on such a scale that I use it on a daily basis.

https://github.blog/2023-09-06-how-to-build-an-enterprise-llm-application-lessons-from-github-copilot/

Speaking for Hackers - public speaking is hard. From choosing a topic, submitting a CFP, preparing your talk and slides, and wrapping it all up. Every step can be tricky, and each of us has other things that are harder for us. This site provides excellent materials for all the parts before, during, and after the talk, making it easier to step out of our shells and share the knowledge.

https://sfhbook.netlify.app/

5 interesting things (09/02/2024)

Fri, 09 Feb 2024 10:21:02 GMT

Closing the women’s health gap: A $1 trillion opportunity to improve lives and economies - a McKinsey report that highlights the gender health gap and points to the opportunity - potential for a $1 trillion economic gain with additional societal impact. One interesting point is that there are gaps and flaws throughout the value chain - drug effectiveness, therapy access, research functions, etc. This hints that there are many opportunities out there that can make a significant impact.

https://www.mckinsey.com/mhi/our-insights/closing-the-womens-health-gap-a-1-trillion-dollar-opportunity-to-improve-lives-and-economies

Slashing Data Transfer Costs in AWS by 99% - one of the costs developers often forget or dismiss when considering architecture is the cost of data transfer. The solution described in this post is elegant and demonstrates the effect of deep knowledge and understanding of the domain. Simple to trivial architectural decisions can cost so much.

https://www.bitsand.cloud/posts/slashing-data-transfer-costs

3 questions that will make you a phenomenal rubber duck - I previously mentioned that debugging skills are essential, and it is important to iterate and refine them. I especially liked the 3rd question - "If your hypothesis were wrong, how could we disprove it?" as it forces one to think the other way around and see a slightly bigger picture.

https://blog.danslimmon.com/2024/01/18/3-questions-that-will-make-you-a-phenomenal-rubber-duck

Product Managing to Prevent Burnout - burnout is more common than we think and can have many causes. Moreover, different people would react differently to different cultures and would burn out or not burn out accordingly. The most important takeaway is that managing and controlling burnout is a team sport; it is not only the concern of the direct manager, but product managers can also participate in this effort. (I strongly recommend the honeycomb blog)

https://www.honeycomb.io/blog/product-managing-prevent-burnout

The “errors” that mean you’re doing it right - I was able to identify or witness almost all the errors mentioned in the post. I also think some of those errors, such as Letting someone go soon after hiring, Pivoting a strategy just after creating it, etc, could be attributed to the sunk cost fallacy. And if we want to make the opening sentence more extreme - "If you don’t make mistakes, you’re not working".

https://longform.asmartbear.com/good-problems-to-have

5 interesting things (15/01/2024)

Mon, 15 Jan 2024 09:58:01 GMT

SQL as API - I saw several efforts to expose RDBMS as API over the years. This post suggests another engel - exposing an API that accepts SQL. Consider this a brain teaser.

https://valentin.willscher.de/posts/sql-api/

SomeEstimates - For me, the loss of trust described in the post is the most harassing implication of a culture where estimates are often missed -

"Another negative outcome is a loss of trust between developers and management since a constant sense of urgency is tantamount to no sense of urgency at all."

https://www.shaiyallin.com/post/someestimates

How to Make Anthropic’s Claude Models Consistently Generate Valid JSON - Gettign valid and consistent JSON from LLM is an issue. Prompt engineering, as described in this post, can solve some of those issues; the json_repair package mentioned there can solve additional problems. With the GPT store announced this week and the evolving models, I believe this will be solved soon in one way or another.

https://levelup.gitconnected.com/how-to-make-anthropics-claude-models-consistently-generate-valid-json-d74ce037ca46

Bonus - https://github.com/mangiucugna/json_repair

My PostgreSQL wishlist - Another brain teaser. The items I most relate to are having created_at and updated_at columns created and maintained automatically and being immutable. I'm curious to follow the comments on this post.

https://ryanguill.com/postgresql/sql/2024/01/08/postgresql-wishlist.html

Everyday storytelling for engineers. The CAO Method - Although storytelling has become an overused buzzword in the last few years (I thought it was already over the hill). This post is important not due to the specific method but to the recognition that ICs practice storytelling every day, and mastering this skill can affect your promotion, career path, tasks you get, etc.

https://tonyfreed.substack.com/p/everyday-storytelling-for-engineers

Cannot get geekier

Sat, 06 Jan 2024 15:09:31 GMT

I contributed my part and created a connections board -

https://connections.swellgarfo.com/game/-NnTsSZ5VzgJeM1OVLDa

5 interesting things (13/12/2023)

Wed, 13 Dec 2023 20:58:57 GMT

Engineering Team Lessons from Cycling - having a background in team sports (Rugby) and individual sports (running), I enjoy such posts that bring experience from one domain to another.

https://benjiweber.co.uk/blog/2023/10/15/engineering-team-lessons-from-cycling/

How to (and how not to) design REST APIs - although I read several posts about REST API best practices, I found this post very insightful, reasoned, and with great examples.

https://github.com/stickfigure/blog/wiki/how-to-(and-how-not-to)-design-rest-apis

Handling a Regional Outage: Comparing the Response From AWS, Azure and GCP - luckily for the post author, all the major cloud services had regional outages in the last while, so he can compare their responses. This will not tip the scales when choosing a cloud provider but will let you know what to expect. It is also an interesting thought on handling outages as a provider.

https://open.substack.com/pub/pragmaticengineer/p/handling-a-regional-outage-comparing

Python Errors As Values - it is sometimes tough to move from one technology to another - being a newbie all over again, thinking differently, adapting to a new ecosystem, etc. It also makes you ponder concepts that were previously perceived as obvious. For example, the approach for errors in Python. Without spoilers - there is an elegant Pythonic way to implement it.

https://www.inngest.com/blog/python-errors-as-values

croniter - this is a cron utilities package. For example, it helps you find the next time a cronjob should be called given a datetime object. It can also find the previous iteration, validate a cron expression, test if a datetime matches a cron condition, etc.

https://github.com/kiorky/croniter

6 Thoughts on Smart Brevity

Tue, 10 Oct 2023 21:57:57 GMT

Another one bites the dust - I just finished hearing "Smart Brevity: The Power of Saying More with Less" by Jim VandeHei, Mike Allen, and Roy Schwartz. One of my chief complaints about engineering training is the dismissal of communication skills - speaking, reading, and writing.

https://twitter.com/tomron696/status/1588241015034388480

Effective communication for developers is everywhere -

Being succinct and accurate in daily meetings
Opening clear tickets so others can understand, reproduce, and prioritize
Effectively communicate their work, suggestions, etc.

Here are some thoughts I had while reading the book -

Using LLMs - chatGPT, along with other tools such as Grammarly, Wordtune, and so on, can help you tune the tone of your passage and adjust it to your audience. The book recommends asking a friend for feedback, but in 2023, we can first ask chatGPT.
CV - the book suggests many scenarios for using smart brevity - social media, presentations, etc. I would also like to suggest writing a CV. Many CVs start with a short paragraph about the person. You can use this paragraph wisely.
Spell out your takeaways - I understand the rationale while struggling with this advice. This is a rhetoric trick - tell your audience what is the one thing you want them to take from this talk instead of leaving it open.
Amazon 6 pagers - the book starts with saying like "words overdose" and "words addicts". Stating that since we moved to the web instead of print, the number of words is unlimited, and nobody reads them besides the headlines. Then, they refer to Amazon 6 pagers as a good example of a compact way of transferring a message. Six pages is a lot, and there is probably a way to reduce the length.
Culture - I find this book very American-centric and complains that people need to be more direct. This is, of course, relative to the Israeli culture I come from. I strongly recommend reading "The Culture Map" by Erin Meyer on this topic.
Content structure - one of the repeating recommendations is to use bullet points to draw attention and help the audience focus. Additional recommendations include using bold fonts to emphasize important points, combining graphs and charts, etc. As a reader, I get this completely. As a writer who tries to distill her words, I want my words to stand out without using those tricks.

https://twitter.com/tomron696/status/1528730823641407488

5 thoughts on Working Backwards

Sat, 07 Oct 2023 17:24:31 GMT

I had a long day of walking around and waiting a lot, so it was a good chance to listen "Working Backwards" by Colin Bryar and Bill Carr. It was more insightful than I anticipated. Here are my thoughts about it -

Corporates vs. startups - whenever I read or hear about best practices and success stories of big organizations (Amazon, Netflix, etc.), I wonder what I can adapt and what I can use for a small startup. Two things sparked my mind -
- Hiring for diversity - the book mentioned that they cared for diversity, specifically concerning gender, from the early days of Amazon. They say that one of the groups noticed that they had some bias in their hiring process, specifically in CV screening, and wanted to improve it while not changing the bar. To eliminate biases that arise when reading women's CVs, they modified the hiring process so that every woman who applied to the position would go through a phone interview.
- Diving into the details - in startups, there are always fires to put away, new requests, and urgent tasks. I can improve on diving into the details, and that can put some of the fires away.
"Be stubborn on the vision but flexible on the details." - I love this quote, and I think it is one of the enablers for Amazon's innovative culture.
Blackberry inspiration - the authors mention that when designing Kindle, they were inspired by Blackberry, which was innovative for the time being as it was always synced and available. As in the examples in the Jobs to Be Done book - while the inspiration stands in place, the example is a bit outdated. Maybe I should read this next.
6 pagers - one of the justifications brought in the book for moving from PowerPoint to six pagers is that it eliminates the differences between a skilled marketing person, a junior developer, and an experienced VP and makes it more equal. I disagree with this argument. It might decrease the dependency on presentation skills, but I still think that there is a gap and experience matters even in writing - choosing the right wording and style and surfacing the relevant doubts, concerns, or benefits of the relevant stakeholders and audience. Writing is also an art, and writing a concise passage is not an easy task.
WatchNow subscription - when talking about Prime Videos, they mentioned Netflix's early days when it was called WatchNow, and they offered a subscription to whoever had a DVD subscription. This resonates with jobs to be done advice regarding obstacles to adaptation - offer the customers a way to try the product (freemium, limited trial, etc) before they buy it.
AWS - as they say, the origin story and other stories about AWS can fill a book independently. I look forward to such a book. Having said that, they talk about the pricing of S3 and whether this should be a subscription or usage-based, what should count for usage - storage size, API calls, etc., and how the pricing was changed once they better understood the usage patterns. This is a great anecdote that even if you walk backward and prepare the PR and Q&A before you develop the product, you will learn new things when users start using it.
2 pizza teams legend - there are a few concepts associated with Amazon, such as the "2 pizza teams" which, after reading the book, I find it is highly misunderstood or very freely interpreted. On the other hand, I didn't hear many companies or many people discussing the "Working Backwards" process, which I find far more interesting, and I wonder why one concept is so popular while another one stays in the shade.

5 thoughts on Jobs to Be Done

Fri, 06 Oct 2023 23:47:31 GMT

After it was mentioned in "Second Brain" and was waiting in my reading list for a long time, I finally listen to "Jobs to Be Done: A Roadmap for Customer-Centered Innovation" by Stephen Wunker, Jessica Wattman, and David Farber. Here are my thoughts about it -

1. Examples relevancy - The book was published in 2016 are brings several examples - slack, snapchat, etc. Are those examples still relevant? Do they still emphasize the relevant points?

This problem is not unique to this book. One often finds an anecdote and uses it to emphasize or justify a theory. As time passes, the perspective also changes, and it sometimes differs from the theory.

2. Multiple stakeholders - B2B purchasing procedures frequently involve a varied group of stakeholders, each with their unique jobs to be done. This point of view is often not thought of and has great implications for how sales should done.

3. Emotional and Social Components - jobs to be done are not only functional tasks such as getting from here to there, wearing something, etc. Jobs to be done also have emotional and social components that should be addressed.

4. Obstacles to use and obstacles to adoption - Obstacles to adoption are challenges that restrict a consumer's inclination to purchase a product or service. Facilitating the ease with which people can learn about and experiment with your new offering can diminish obstacles to adoption. Obstacles to use refer to impediments that hinder success, ultimately reducing a customer's probability of ongoing product usage, acquiring supplementary features, or upgrading to more recent editions. I love this insight, and it is an important distinction, especially when one needs to prioritize.

5. Effective Brainstorming - brainstorming is discussed in long in one of the chapters. Personally, I have many doubts about group brainstorming as a way to encourage creativity. If there is one thing to take from this discussion, such a session should be well-mederated. See more here.

5 thoughts on Building a Second Brain

Wed, 27 Sep 2023 08:22:34 GMT

"a wealth of information creates a poverty of attention and a need to allocate that attention efficiently among the overabundance of information sources that might consume it.”
Herbert A. Simon

The quote above appears in "Building a Second Brain" by Tiago Forte, which I finished reading last week. I strongly relate to it, and that book helps me reflect on my personal knowledge management.

TL;DR - find a personal knowledge management method that works for you (e.g., the PARA method). It does not have to be perfect. You don't have to shift everything. Just get it started and adjust as you go.

Here are a few thoughts I had while reading the book -

1. Progressive Summarization - the progressive summarization technique reminded me of a joke my uncle told me a while back - a student in his first semester asked the lecturer how to prepare best for the exam. She tells him - after every class, summarize your class notes. At the end of each week, summarize all the daily summaries. At the end of each month, summarize the weekly summaries and so on. On the week before the exam, summarize the summary from the day before. They meet just before the exam, and she asks him how it went. He answers - "great, I was able to summarize everything into one word - bullshit."

2. Divergence and Convergence - In my first or second semester in the university, I took a class on academic writing. They told us that a good academic essay is built like an hourglass. It starts with a very wide question or statement, then narrows down to a specific claim or private case, and finishes with the broader picture, zoom out, etc. See more here. Divergence and convergence are the same. You start very scattered, then connect the dots, focus, reach some advancement, and repeat.

3. Hemingway's bridges - "The “Hemingway Bridge” is a technique used by author Ernest Hemingway in which he would stop his writing for the day only AFTER he knew what was coming next." (here). Each of us has its own hooks that help him or her restart the next time. A few years ago, I read Hila Noga's post about getting your programming flow going, and it is Hemingway's bridge for developers.

4. Blog as an interface - I initially created the blog so it would be easier for me to search for links I once saw and to share with other people. One can view it as some interface to my second brain. I am still in the process of thinking about which methods are right for me to adapt from the book.

5. Listening to an e-book - I'm a big fan of highlighting and writing comments in books, papers, etc. The audiobook format is challenging for me in this aspect, and moreover, I usually listen to an e-book while doing other things like walking or driving, which misses some of the second brain practice. I still need to figure out how to tackle this. On the other hand, I use writing and notes of all kinds to unload my brain and as an easier way to access them in the future. I was very happy that the topic of offloading was widely discussed in the book.

Team Health Check

Thu, 07 Sep 2023 17:12:00 GMT

Today, I heard Dafna Rosenblum’s (see Dafna’s blog here) talk on the EMIL (Engineering Manager IL) meetup about “Team Health Check”. It was the first time I heard about the concept, so I read more about it.

Spotify developed the team health check concept and introduced it in 2014 (here). About 6 months ago, Spotify published a new post about “Getting More from Your Team Health Checks”. The post focuses on improving the team experience in this workshop and suggests the following main ideas -

Customize wisely - tailor the right questions and health checks that fit the team and the organization.
Dig Deep - Good facilitation is essential to enable more profound conversations. Good facilitation should help to step outside the team's day-to-day communication patterns and create a psychologically safe place (see here) to raise issues. The post suggests a few strategies on how to do it.
Follow through - you should follow up and reiterate the topics that were raised in the team health check workshop. That can be in 1-1 meetings, scheduling required meetings, checking your priority or attention, etc.

I found a few online tools to help facilitate team health checks -

teamhealthcheck.io - “A free, anonymous, and super simple tool to run a version of the Spotify Health Check Survey. “
Miro boards - There are multiple miro templates for team health checks - e.g., here, here, and here.

Another tip from Dafna to start meeting (and every meeting) is to start with a quick intake to break the ice and increase engagement. For example, ask the member to describe their mood using an emoji.

5 interesting things (04/09/2023)

Mon, 04 Sep 2023 10:08:33 GMT

12 Debugging tools I wish I knew earlier - it describes more debugging strategies than debugging tools (i.e. minimal reproduction is not a tool). One strategy I missed in this post is adding breakpoints. If I were to write this post, I would order it in an escalation order. For example, reading the error message would be in a higher place. However, it is an important post, especially for junior developers.

https://careercutler.substack.com/p/12-debugging-tools-i-wish-i-knew

Consistency Patterns - This post explains the different common consistency patterns - strong, eventual consistency, and weak consistency and the trade-offs. It also mentions the idea of causal consistency, which I find very interesting.

https://systemdesign.one/consistency-patterns/

Remote work requires communicating more, less frequently - he had me at "Think of it like gzip compression, but for human-to-human communication. Yes, there’s slightly more processing overhead at the start, but it allows greater communications throughput using fewer “packets” (communicate more using less)". Seriously, once your organization grows above ten people and you start having clients, you will have people remote (colleagues or clients), and you will have to optimize your communication to pass your message.

https://ben.balter.com/2023/08/04/remote-work-communicate-more-with-less/

Git log customization - I'm setting a new computer now for development and looking for a format that would be easy for me to use so this post came exactly on time

https://www.justinjoyce.dev/customizing-git-log-format/

Structuring your Infrastructure as Code - I like the layers approach of this post and the examples from all 3 public cloud providers. I would like to give more thought to the exact layers and order. Note that this post is written by Pulumi, a solution engineer, so it might not work well with other IaC tools.

https://leebriggs.co.uk/blog/2023/08/17/structuring-iac

5 interesting things (27/07/2023)

Thu, 27 Jul 2023 19:54:56 GMT

Designing Age-Inclusive Products: Guidelines And Best Practices - I have a 91-year-old grandmother who, in the last 10 years, cannot book a doctor's appointment herself as she does not use a smartphone and cannot follow voice navigation. Even without a personal perspective, I am very interested in accessibility, and I try to pay attention to inclusivity and accessibility topics wherever relevant. However, I always wonder if those are general best practices or are limited to specific cohorts. Specifically, in this case, younger people usually have more technology literacy than older people and therefore can achieve their goals with less optimized flows and UI.

https://www.smashingmagazine.com/2023/07/designing-age-inclusive-products-guidelines-best-practices/

On Becoming VP of Engineering - A two-part blog post series by Emily Nakashima, Honeycomb’s first VP of Engineering. The first part focuses on her path - coming originally from design, frontend, and product engineering and becoming VP of Engineering that also manages the backend and infrastructure.

The second part talks about the day-to-day work and the shift in focus when moving from a director position to a VP position. I strongly agree with her saying, "Alignment is your most important deliverable," and also think it is one of the hardest things to achieve.

https://www.honeycomb.io/blog/becoming-vp-of-engineering-pt1

https://www.honeycomb.io/blog/becoming-vp-of-engineering-pt2

Project Management for Software Engineers - "This article is a collection of techniques I’ve learned for managing projects over time, that attempts to combine agile best practices with project management best practices.". While a degree in computer science teaches lots of algorithms, software development, and so on, it does not teach project management and time management. Those skills are usually not required in junior positions but can help you have a more significant impact. Having said that, one should find the exact practices that fit him or her and that can evolve over time.

https://sookocheff.com/post/engineering-management/project-management-for-software-engineers/

Designing Pythonic library APIs - A while ago (2 years+-), I looked for a post/tutorial / etc. regarding designing SDK best practices and could not find something I was happy with. I like the examples (both good and bad examples) in this post. If you are in a hurry, all the take aways are summarized in the end (but sometimes hard to understand without context).

https://benhoyt.com/writings/python-api-design/

Fern - "Fern is an open source toolkit for designing, building, and consuming REST APIs. With Fern, you can generate client libraries, API documentation, and boilerplate for your backend server.". I haven't tried it myself yet, but if it works, it seems like cookie-cutter on steroids. In the era of LLMs, the next step is to generate all of those from free text.

https://github.com/fern-api/fern

5 interesting things (06/07/2023)

Thu, 06 Jul 2023 13:55:30 GMT

Potential impacts of Large Language Models on Engineering Management - this post is an essential starter for a discussion, and I can think of other impacts. For example - how interviewing \ assessing skills of new team members affected by LLMs? What skills should be evaluated those days (focusing on engineering positions)?

One general caveat for using LLMs is completely trusting them without any doubts. This is crucial for a performance review. Compared to code, if the code does not work, it is easy to trace and fix. If the performance review needs to be corrected, it might be hard to pinpoint what and where it got wrong, and the person getting it might need more confidence to say something.

https://www.engstuff.dev/p/potential-impacts-of-large-language

FastAPI best practices - one of the most reasoned and detailed guides I read. Also, the issues serve as comments to this guide and are worth reading. Ideally, I would like to take most of the ideas and turn them into a cookie-cutter project that is easy to create.

https://github.com/zhanymkanov/fastapi-best-practices

How Product Strategy Fails in the Real World — What to Avoid When Building Highly-Technical Products - I saw all in action and hope to do better in the future.

https://review.firstround.com/how-product-strategy-fails-in-the-real-world-what-to-avoid-when-building-highly-technical-products

1 dataset 100 visualizations - I imagine this project as an assignment in a data visualization/data journalism course. Yes, there are many ways to display data. Are they all good? Do they convey the desired message?

There is a risk in being too creative, and there is some visualization there I cannot imagine using for anything reasonable.

https://100.datavizproject.com/

Automating Python code quality - one additional advantage of using tools like Black, isort, etc., is that it reduces the cognitive load when doing a code review. The code reviewer should no longer check for style issues and can focus on deeper issues.

https://blog.fidelramos.net/software/python-code-quality

Bonus - more extensive pre-commit template -

https://github.com/br3ndonland/template-python/blob/main/.pre-commit-config.yaml

Did you Miss me? PyCon IL 2023

Tue, 04 Jul 2023 16:20:00 GMT

Today I talked about working with missing data at PyCon IL. We started with a bit of theory about mechanisms of missing data -

MCAR - The fact that the data are missing is independent of the observed and unobserved data.
MAR - The fact that the data are missing is systematically related to the observed but not the unobserved data.
MNAR - The fact that the data are missing is systematically related to the unobserved data.

And deep-dived into an almost real-world example that utilizes the Python ecosystem - pandas, scikit-learn, and missingno.

My slides are available here and my code is here.

3 related posts I wrote about working with missing data in Python -

Pandas fillna vs scikit-learn SimpleImputer
pandas read_csv and missing values
Missing data in Python – 5 resources

Pandas fillna vs scikit-learn SimpleImputer

Wed, 21 Jun 2023 11:24:40 GMT

Missing data is prevalent in real-world data and can be missing for various reasons. Gladly, both pandas and scikit-learn several imputation tools to deal with it. Pandas offers a basic yet powerful interface for univariate imputations using fillna and more advanced functionality using interpolate. scikit-learn offers both SimpleImputer for univariate imputations and KNNImputer and IterativeImputer for multivariate imputations. In this post, we will focus on fillna and SimpleImputer functionality and compare them.

Basic Functionality

SimpleImputer offers four strategies to fill in the nan values - mean, median, most_frequet, and constant.

import numpy as np
import pandas as pd
from sklearn.impute import SimpleImputer

df = pd.DataFrame(
    [[7, 2, np.nan], [4, np.nan, 6], [10, 5, 9]])
imp_mean = SimpleImputer(strategy='mean')
pd.DataFrame(imp_mean.fit_transform(df))

output -

      0    1    2
0   7.0  2.0  7.5
1   4.0  3.5  6.0
2  10.0  5.0  9.0

Can we achieve the same with pandas? Yes!

df.fillna(df.mean())

Want to impute with the most frequent value?

Asuume - df = pd.DataFrame(['a', 'a', 'b', np.nan])

With SimpleImputer -

imp_mode = SimpleImputer(
    strategy='most_frequent')
pd.DataFrame(
  
  imp_mode.fit_transform(df))

With fillna -

df.fillna(df.mode()[0])

And the output of both -

Different Strategies

Want to apply different strategies for different columns? using scikit-learn you will need several imputers, one per each strategy. Using fillna you can pass a dictionary, for example -

df = pd.DataFrame(
    [[7, 2, np.nan], [4, np.nan, 6], [10, 5, 9]])
df.fillna({1: 10000, 2: df[2].mean()})

    0        1    2
0   7      2.0  7.5
1   4  10000.0  6.0
2  10      5.0  9.0

Advanced Usage

Want to impute values drawn from a normal distribution, no brainer -

mean = 5
scale = 2
df = pd.DataFrame(
    [[7, 2, np.nan], [4, np.nan, 6], [10, 5, 9]])
df.fillna(
    pd.DataFrame(
        (np.random.normal(mean, scale, df.shape))

    0         1         2
0   7  2.000000  3.857513
1   4  5.407452  6.000000
2  10  5.000000  9.000000

Missing indicator

Using SimpleImputer, one can add indicator columns that obtain 1 if the original column was missing, and 0 otherwise. This can also be done using MissingIndicator

df = pd.DataFrame(
    [[7, 2, np.nan], [4, np.nan, 6], [10, 5, 9]])
mean_imp = SimpleImputer(strategy='mean', add_indicator=True)
mean_imp.fit_transform(df)
pd.DataFrame(mean_imp.fit_transform(df))

      0    1    2    3    4
0   7.0  2.0  7.5  0.0  1.0
1   4.0  3.5  6.0  1.0  0.0
2  10.0  5.0  9.0  0.0  0.0

Note that a missing column (i.e., columns 3 and 4 in the example above) corresponds only to columns with missing values. Therefore there is no missing indicator column corresponding to the column 0. If you are converting back and forth to pandas dataframes you should note this nuance.

Another nuance to note when working with SimpleImputer is that columns that contain only missing values are dropped by default -

df =  pd.DataFrame(
    [[7, 2, np.nan, np.nan], [4, np.nan, 6, np.nan],
    [10, 5, 9, np.nan]])
mean_imp = SimpleImputer(strategy='mean')
pd.DataFrame(mean_imp.fit_transform(df))

      0    1    2
0   7.0  2.0  7.5
1   4.0  3.5  6.0
2  10.0  5.0  9.0

This behavior is controllable using setting keep_empty_features=True. While it is manageable, tracing columns might be challenging -

mean_imp = SimpleImputer(
    strategy='mean',
    keep_empty_features=True,
    add_indicator=True)
pd.DataFrame(mean_imp.fit_transform(df))

      0    1    2    3    4    5    6
0   7.0  2.0  7.5  0.0  0.0  1.0  1.0
1   4.0  3.5  6.0  0.0  1.0  0.0  1.0
2  10.0  5.0  9.0  0.0  0.0  0.0  1.0

There is an elegant way to achieve similar behavior in pandas -

df = pd.DataFrame(
    [[7, 2, np.nan, np.nan], [4, np.nan, 6, np.nan],
     [10, 5, 9, np.nan]])
pd.concat(
    [df.fillna(df.mean()), 
     df.isnull().astype(int).add_suffix("_ind")], axis=1)

    0    1    2   3  0_ind  1_ind  2_ind  3_ind
0   7  2.0  7.5 NaN      0      0      1      1
1   4  3.5  6.0 NaN      0      1      0      1
2  10  5.0  9.0 NaN      0      0      0      1

Working with dates

Want to work with dates and fill several columns with different types? No problem with pandas -

df = pd.DataFrame(
    {"date": [
        datetime(2023, 6, 20), np.nan,
        datetime(2023, 6, 18), datetime(2023, 6, 16)],
     "values": [np.nan, 1, 3, np.nan]})
df.fillna(df.mean())

Before -

        date  values
0 2023-06-20     NaN
1        NaT     1.0
2 2023-06-18     3.0
3 2023-06-16     NaN

After -

        date  values
0 2023-06-20     2.0
1 2023-06-18     1.0
2 2023-06-18     3.0
3 2023-06-16     2.0

Working with dates is an advantage that fillna has over SimpleImputer.

Backward and forward filling

So far, we treated the records and their order as independent. That is, we could have shuffled the records and that would not affect the expected imputed value. However, there are cases, for example, when representing time series when the order matters and we would like to impute based on later values (backfill) or earlier values (forward fill). This is done by setting the method property.

df = pd.DataFrame(
    [[7, 2, np.nan], [4, np.nan, 6],
     [10, np.nan, 9], [np.nan, 5, 10]])
df.fillna(method='bfill')

      0    1     2
0   7.0  2.0   6.0
1   4.0  5.0   6.0
2  10.0  5.0   9.0
3   NaN  5.0  10.0

One can also limit the number of consecutive values which are imputed -

df.fillna(method='bfill', limit=1)

      0    1     2
0   7.0  2.0   6.0
1   4.0  NaN   6.0
2  10.0  5.0   9.0
3   NaN  5.0  10.0

Note that when using bfill or ffill and moreover, when specifying limit to value other than None it is possible that not all the values would be imputed.

For me, that's a killer feature of fillna comparing to SimpleImputer

Treat Infinite values as na

Setting pd.options.mode.use_inf_as_na = True will treat infinite values (i.e. np.inf, np.INF, np.NINF) values as missing values, for example -

df = pd.DataFrame([1, 2, np.inf, np.nan])
df.fillna(1000)

pd.options.mode.use_inf_as_na = False

pd.options.mode.use_inf_as_na = True

Note that inf and na are not treated the same for other use cases, e.g. - df[0].value_counts(dropna=False)-

0
1.0    1
2.0    1
NaN    1
NaN    1

Summary

Both pandas and scikit-learn offer a basic functionality to deal with missing values. Assuming you are working with pandas Dataframe, pandas fillna functionality can achieve everything SimpleImputer can do and more - working with dates, back and forward fill, etc. Additionally, there are some edge cases and specific behaviors to pay attention to when choosing what to use. For example when using bfill or ffill method some values may not be imputed if there are the last ones or first ones respectively.

Few thoughts on Cloud FinOps Book

Mon, 05 Jun 2023 08:29:42 GMT

I just completed “Cloud FinOps” book by J.R. Storment and Mike Fuller, and here are a few thoughts -

At first, I wondered whether I should read the 1st edition, which I had easy access to, or the 2nd, which I had to buy. After reading a sample, I decided to buy the 2nd edition and am glad. This domain and community move quickly; a 2019 version would have been outdated and misleading.
FinOps involves a paradigm shift - developers should consider not only the performance of their architecture (i.e., memory and CPU consumption, speed, etc.) but the cost associated with the resources they will use. Procurement is not done and approved by the finance team anymore. Developers’ decisions can have a significant influence on the cloud bill. FinOps teams bridge the engineering and finance teams (and more) and speak the language of all parties, along with additional skill sets and an overview of the entire organization.
A general rule of thumb regarding commitments -
1. Longer commitment period (3 years → 1 year) = lower price (higher discount)
2. More upfront (full upfront → partial upfront → no upfront )= lower price (higher discount)
3. More specific (RI → Convertible RI → SP, region, etc.) = lower price (higher discount)
The FinOps team should be up-to-date about the new cloud technologies updates and cost reduction options. I have been familiar with reserve and spot instances for a long time, but there are many other cost reduction options bits and bytes to pay attention to. For example, the following 2 points -
1. When purchasing saving plans (SP), which are monetary as appose to resource units commitments, the spend amount you commit to is post discount. Moreover, AWS will apply the SP to the resources that yield the highest discount. This implies that the discount rate diminishes when committing to more money.
2. CloudFront security savings bundle (here) is a saving plan that ties together the usage of CloudFront and WAF. The book predicts that such plans, e.g., combining multiple product usage, will become common soon.
Commitments (e.g., SP, RI) are one of many ways to reduce costs. Removing idle resources (e.g., unattached drives), using correct storage classes (e.g., infrequent access, glacier), or making architecture changes (e.g., rightsizing, moving from servers to serverless, going via VPC endpoints, etc.) can help avoid and reduce cost. Those activities can happen in parallel - centralized FinOps team to manage commitments (aka cost reduction) and decentralized engineering teams optimize the resources they use (aka cost avoidance). Ideally, it is a tango. Each team moves a little step at a time to optimize their part.
The FinOps domain-specific knowledge goes even further. For example, costs that engineers tend to miss or wrongly estimate e.g. network traffic cost, number of events, data storage events.
The inform phase is part of the FinOps lifecycle - making the data available to the relevant participants. The Prius effect, i.e., real-time feedback, instantly influences behavior even without explicit recommendations or guidance. Visualizations (done right) can help understand and react to the data better. A point emphasized multiple times in the book - put the data in the path of the engineers or any other stakeholder. Don’t ask them to log in to a different system to review the data; integrate with existing systems they use regularly.

Few resources I find helpful -

FinOps foundation website - includes many resources and community knowledge - https://www.finops.org/introduction/what-is-finops/

FinOps podcast - https://www.finops.org/community/finops-podcast/
Infracost lets engineers see a cost breakdown and understand costs before making changes in the terminal, VS Code, or pull requests. https://www.infracost.io/
Cloud Custodian - “Cloud Custodian is a tool that unifies the dozens of tools and scripts most organizations use for managing their public cloud accounts into one open source tool” - https://cloudcustodian.io/
FinOut - A Holistic Cost Management Solution For Your Cloud. I recently participated in a demo and that looks super interesting. https://www.finout.io/
Startup guide to data cost optimization - my post summarizing AWS’s ebook about data cost optimization for startups - https://tomron.net/2023/06/01/startup-guide-to-data-cost-optimization-summary/
Twitter thread I wrote in Hebrew about the book - https://twitter.com/tomron696/status/1657686198327062529

Startup guide to data cost optimization - summary

Thu, 01 Jun 2023 12:44:29 GMT

I read a lot about FinOps and cloud cost optimization those days and I came across AWS short ebook about data cost optimization.

Cost optimization is part of AWS’s well-architected framework. When we think about cost optimization, we usually only consider computing resources, while there are significant optimizations that can go beyond that - storage optimization, network, etc.

Below is a combination of the six sections that appear in the e-books with some comments -

Optimize the cost of information infrastructure - the main point in this section is to use Graviton instances where applicable.

Decouple storage data from compute data - 5 suggestions here which are pretty standard -

Compress data when applicable, and use optimal data structures for your task.
Consider data temperature when choosing data store and storage class - use the suitable s3 storage class and manage it using a life-cycle policy.
Use low-cost compute resources, such as Spot Instances, when applicable - I have some dissonance here since I’m not sure that spot instances are attractive those days (see here), specifically with the overhead of taking care of preempted instances.
Deploy compute close to data to reduce data transfer costs - trivial.
Use Amazon S3 Select and Amazon S3 Glacier Select to reduce data retrieval - Amazon S3 Select has several limitations (see here), so I’m not sure it is worth the effort and better query via Athena.

Plan and provision capacity for predictable workload usage

Choosing the right instance type based on workload pattern and growth - is common sense. You'll save a little less if you purchase convertible reserve instances. However, in a fast-changing startup environment, there is a higher chance the commitment won’t be underutilized.
Deploying rightsizing based on average or medium workload usage - this contradicts best practices described in Cloud FinOps book, so I’m a bit hesitant here.
Using automatic scaling capabilities to meet peak demand - is the most relevant advice in this section. Use auto-scaling groups or similar to accommodate for both performance and cost.

Access capacity on demand for unpredictable workloads

Use Amazon Athena for ad hoc SQL workloads - as mentioned above, I prefer Athena over AWS S3 Select.
Use AWS Glue instead of Amazon EMR for infrequent ETL jobs - I don’t have a strong opinion here, but if you have a data strategy in mind, I will try to adjust to it. Additionally, I feel that other AWS can be even easier and cost-effective to work with—for example, Apache Spark in Amazon Athena, step functions, etc.
Use on-demand resources for transient workloads or short-term development and testing needs - having said that, you should still keep an eye on your production services, ensure they are utilized correctly and rightsize them if needed.

Avoid data duplication with a centralized storage layer

Implement a central storage layer to share data among tenants - I would shorten it to saying, “have a data strategy” - where you are, where you want to go, etc., which is not trivial in early startup days.

Leverage up to $100,000 in AWS Activate credits

This might be a bit contracting to the rest of the document since it feels like free money and delays your concern about cloud costs.

5 interesting things (16/05/2023)

Tue, 16 May 2023 13:58:16 GMT

Women’s health research lacks funding – these charts show how - not a proper tech link but - I liked the infographic very much (it missed some hovering features) and believe this is an important topic.

https://www.nature.com/immersive/d41586-023-01475-2/index.html

Farewell to the Era of Cheap EC2 Spot Instances - spot instances were the holy grail of cloud cost reduction and required a suitable architecture to accommodate it. While the cloud vendors suggest more and more ways to reduce cost, this well seems to dry out and it is backed with data about 5.5 million spot instances they spun over almost seven months. I don't know if it is the end of spot instances, but something goes on.

https://pauley.me/post/2023/spot-price-trends/

Uptime Guarantees — A Pragmatic Perspective - great down-to-earth analysis of uptime and the meaning of each additional nine -

https://world.hey.com/itzy/uptime-guarantees-a-pragmatic-perspective-736d7ea4

Evidence - Business Intelligence as Code - this project intrigued me. Developers often struggle with creating visualizations, the UI of most of the tools is confusing and complex for sporadic use, maybe evidence will unleash it for developers -

https://github.com/evidence-dev/evidence

How to Debug - "The missing Semester of your CS education" (here) influenced how I think of juniors and recently graduated employees. Debugging is a skill you usually don't learn during formal studies and is essential in the industry. This post is a good starting point in the journey of debugging -

https://philbooth.me/blog/how-to-debug

5 interesting things (25/04/2023)

Tue, 25 Apr 2023 10:19:54 GMT

Load balancing - excellent explanations and visualizations about load balancing and different approaches. I wish for follow-up posts about caching and stickiness that influence performance and practical setups - how to set loaded balancers in AWS under those considerations.

https://samwho.dev/load-balancing/

visitdata - A terminal interface for exploring and arranging tabular data. I played with this tool a bit, it is very promising and, at the same time, has a stiff learning curve (think vi) that might keep people away.

https://www.visidata.org/

Software accessibility for users with Attention Deficit Disorder (ADHD) - software accessibility is a topic that I always try to keep in mind. The usual software accessibility patterns refer to visual impairment, e.g., color contrast, font size, etc. This post tackles the accessibility topic from the prism users with ADHD, and I find it groundbreaking. I find that the suggested patterns (e.g., recently opened subscription reminders, etc.) are primarily suitable UX for all users, not just those with ADHD.

https://uxdesign.cc/software-accessibility-for-users-with-attention-deficit-disorder-adhd-f32226e6037c

Minimum Viable Process - I liked the post very much and the following point was the one I relate the most to - Minimum Viable Process process is iterative - processes and procedures must be constantly refined. Processes should evolve along with the company and serve the company rather then the company serve the process.

https://mollyg.substack.com/p/minimum-viable-process

Interactive Calendar Heatmaps with Python — The Easiest Way You’ll Find - always wanted to create a GitHub-like activity visualization? Great, use plotly-calplot for that. See the example here -

https://python.plainenglish.io/interactive-calendar-heatmaps-with-plotly-the-easieast-way-youll-find-5fc322125db7

Thoughts on Tech Debt

Sun, 26 Feb 2023 13:18:40 GMT

I have been thinking about tech debt for a while now and how to address it daily. Few dilemmas for example -

Should we update our Python version or a version of one of the packages we use?
We thought of a more efficient implementation for one of our functions. Should we invest time in it? When?
What task should we prioritize for the next sprint?

How do we measure our technical debt? Are we getting better, and what does better means? Or at least not worse?

So I have compiled a small reading list that I can share with my team and align on terminology and ideas. Feel free to add your thoughts.

The 4 main types of technical debt - this is an explanation of Martin Fowler's Technical Debt Quadrant. This is a first step towards establishing a common language about technical debt.

https://blog.codacy.com/4-types-technical-debt/

The different types of technical debt - Another piece in the puzzle to use a common terminology. While the split into categories makes sense, I don't entirely agree with the fixes and impacts.

https://techdebtguide.com/types-of-technical-debt

The 25 Percent Rule for Tackling Technical Debt - a post from Shopify engineering blog about their pillars of tech debt and a recommendation on how to invest time on those topics. That made me think about whether the discussion about tech debt should differ between different stages or sizes of companies

https://shopify.engineering/technical-debt-25-percent-rule

A Framework for Prioritizing Tech Debt - this post tries to get the analogy closer to real debt and examine the interest rate. If I took this analogy one step further, I would say that it is a loan, and the prioritization and the remediations are the terms that we choose on loan, and sometimes those terms change either because we recycle the loan, win a big sum and can pay back, etc. Or because something external changed, e.g., interest rate.

https://www.maxcountryman.com/articles/a-framework-for-prioritizing-tech-debt

Tech Debt Isn't a Burden, It's a Strategic Lever for Success - the approach here is closer to the loan analogy - "view tech debt as a strategic lever for your organization's success over time". That, together with other points in this post, made me think about the relations and interactions of product debt and tech debt - are they correlated or independent, and what influences them? are there any patterns in this duo or maybe trio (with business debt). I didn't yet find something to read about the topic that I was happy with.

https://www.reforge.com/blog/managing-tech-debt

7 Top Metrics for Measuring Your Technical Debt - this post suggests several metrics to measure technical debt, such as code churn, cycle time, code quality, and tools for measuring technical debt. One issue is that it does not cover infrastructure debt or process debt. For example - the complexity of deploy process, duplication between repositories, etc. Additionally, the underlying assumption in this post is that "tech debt is bad," while I view it as a strategy or a trade-off. I also don't believe that one size fits all - if you want to measure, choose the one thing that is most important and informative for you, and don't worry if it changes over time.

https://dev.to/alexomeyer/8-top-metrics-for-measuring-your-technical-debt-5bnm

Exploratory Data Analysis Course - Draft

Fri, 30 Dec 2022 11:27:27 GMT

Last week I gave an extended version of my talk about box plots in Noa Cohen's Introduction to Data Science class at Azrieli College of Engineering Jerusalem. Slides can be found here.

The students are 3rd and 4th-year students, and some will become data scientists and analysts. Their questions and comments and my experience with junior data analysts made me understand that a big gap they have in purchasing those positions and performing well is doing EDA - exploratory data analysis. This reminded me of the missing semester of your CS education - skills that are needed and sometimes perceived as common knowledge in the industry but are not taught or talked about in academia.

"Exploratory Data Analysis (EDA) is the crucial process of using summary statistics and graphical representations to perform preliminary investigations on data in order to uncover patterns, detect anomalies, test hypotheses, and verify assumptions." (see more here). EDA plays an important role in everyday life of anyone working with data - data scientists, analysts, and data engineers. It is often also relevant for managers and developers to solve the issues they face better and more efficiently and to communicate their work and findings.

I started rolling in my head how would a EDA course would look like -

Module 1 - Back to basics (3 weeks)

Data types of variables, types of data
Basic statistics and probability, correlation
Anscombe's quartet
Hands on lab - Python basics (pandas, numpy, etc.)

Module 2 - Data visualization (3 weeks)

Basic data visualizations and when to use them - pie chart, bar charts, etc.
Theory of graphical representation (e.g Grammar of graphics or something more up-to-date about human perception)
Beautiful lies - graphical caveats (e.g. box plot)
Hands-on lab - python data visualization packages (matplotlib, plotly, etc.).

Module 3 - Working with non-tabular data (4 weeks)

Data exploration on textual data
Time series - anomaly detection
Data exploration on images

Module 4 - Missing data (2 weeks)

Missing data patterns
Imputations

Hands-on lab - a combination of missing data \ non-tabular data

Extras if time allows-

Working with unbalanced data
Algorithmic fairness and biases
Data exploration on graph data

I'm very open to exploring and discussing this topic more. Feel free to reach out - twitter, LinkedIn.

5 interesting things (03/11/2022)

Thu, 03 Nov 2022 20:52:01 GMT

How to communicate effectively as a developer. - writing effectively is the second most important skill after reading effectively and one of the skills that can differentiate you and push you forward. If you read only one thing today, read this -

https://www.karlsutt.com/articles/communicating-effectively-as-a-developer/

26 AWS Security Best Practices to Adopt in Production - this is a periodic reminder to pay attention to our SecOps. This post is very well written and the initial table of AWS security best practices by service is great.

https://sysdig.com/blog/26-aws-security-best-practices/

EVA Video Analytics System - "EVA is a new database system tailored for video analytics -- think MySQL for videos.". Looks cool on first glance and I can think off use cases for myself, yet I wonder if it could become a production-level grade.

https://github.com/georgia-tech-db/eva

I see it as somehow complementary to - https://github.com/impira/docquery

Forestplot - "This package makes publication-ready forest plots easy to make out-of-the-box.". I like it when academia and technology meet and this is really usable, also for data scientists' day-to-day work. The next step would probably be deep integration with scikit-learn to pandas.

https://github.com/lsys/forestplot

Bonus - Python DataViz cookbook - easy way to navigate between the different common python visualization practices (i.e via pandas vs using matplotlib / plotly / seaborn). I would like to see it going to the next step - controlling the colors, grid, etc. from the UI and then switching between the frameworks but that's a starting point.

https://dataviz.dylancastillo.co/

roadmap.sh - it is not always clear how to level up your skills, what you should learn next (best practices, technology - which, etc). Roadmap.sh attempts to create such roadmaps. While I don't agree with everything there, I think that the format and references are nice and it is a good inspiration.

https://roadmap.sh/

Shameless plug - Growing A Python Developer (2021), I plan to write a small update in the near future.

Think outside of the Box Plot

Thu, 15 Sep 2022 20:25:05 GMT

Earlier today, I spoke at DataTLV conference about box plots - what they expose, what they hide, and how they mislead. My slides can be found here, and the code used to generate the plots is here.

Key takeaways

Boxplots show 5 number statistics - min, max, median, q1 and,q3.
The flaws of Box Plots can be divided into two - data that is not present in the visualization (e.g. number of samples, distribution) and the visualization being counter-intuitive (e.g. quartiles is hard to grasp the concept).
I choose solutions that are easy to implement, either by leveraging existing packages code or by adding small tweaks. I used plotly.
Aside of those adjustment I many times box plot is just not the right graph for the job.
If the statistical literacy of your audience is not well founded I would try avoiding using box plot.

Topics I didn't talk about and worth mentioning

Mary Eleanor Hunt Spear - data visualization specialize who pioneered the development of the bar chart and box plot. I had a slide about her but went too fast, and skipped it. See here.
How percentiles are calculated - Several methods exist, and different Python packages use different default methods. Read more -http://jse.amstat.org/v14n3/langford.html

Resources I used to prepare the talk

https://blog.minitab.com/en/statistics-and-quality-data-analysis/how-to-think-outside-the-boxplot
https://www.statisticshowto.com/probability-and-statistics/descriptive-statistics/box-plot/
https://asklexph.com/thinking-outside-the-box-plot
https://nightingaledvs.com/ive-stopped-using-box-plots-should-you/
https://www.greenbook.org/mr/market-research-news/replacing-boxplots-and-histograms-with-rugs-violins-and-bean-plots/

5 interesting things (29/08/2022)

Mon, 29 Aug 2022 20:49:31 GMT

Human genetics 101 - a new blog about genetics by Nadav Brandes, who works at UCSF as part of the Ye lab. Reading is very accessible even to non-biologist (like me :).

https://incrementally.net/2022/07/16/human-genetics-101/

It's probably time to stop recommending Clean Code - that's a relatively old post (from 2020) discussing a book that was published in 2008. It is a very common recommendation in the industry, and therefore, I think this post is so important. It is detailed and gives good examples, and reminds us that everything has to be taken with a grain of salt. I agree with the concluding paragraphs - experienced developers will gain almost nothing from reading the book, and inexperienced developers would have a hard time separating the wheat from the chaff.

https://qntm.org/clean

Bonus - https://gordonc.bearblog.dev/dry-most-over-rated-programming-principle/

The many flavors of hashing - I like to be back to basic from time to time.

https://notes.volution.ro/v1/2022/07/notes/1290a79c/

Five Lessons Learned From Non-Profit Management That Apply to Tech Management - I like those mixes when practices and ideas from one domain of someone's life emerge in another domain. Those intersections are usually very productive and insightful.

https://medium.com/management-matters/5-lessons-learned-from-non-profit-management-that-apply-to-tech-management-add47980498a

Demystifying the Parquet File Format - I finally feel I understand how the parquet format works (although there are many more optimizations).

https://towardsdatascience.com/demystifying-the-parquet-file-format-13adb0206705

5 interesting things (22/07/2022)

Fri, 22 Jul 2022 08:37:38 GMT

I analyzed 1835 hospital price lists so you didn't have to - this post had a few interesting things. First, learning about CMS's price transparency law. In Israel this is a non-issue since the healthcare system works differently, and most of the procedures are covered by the HMOs so there is no such concern. I would be interested in further analysis about the missing or non-missing prices. I.e., for which CPT codes most hospitals have prices, for which CPT codes most hospitals don't have prices, can we cluster them (e.g. cardio codes? women's health? procedures usually done on elder people?). This dataset has great potential, and I agree with most of the points in the "Dead On Arrival: What the CMS law got wrong" section.

https://www.dolthub.com/blog/2022-07-01-hospitals-compliance/

How to design better APIs - there are several things I liked in this post - first, it is written very clearly and gives both positive and negative examples. Second, it is language agnostic. That last tip - "Allow expanding resources" was mind-blowing to me, so simple to think of and I never thought of adding such an argument. Now I miss a cookie-cutter template to implement all that good advice.

https://r.bluethl.net/how-to-design-better-apis

min(DALL-E) - "This is a fast, minimal port of Boris Dayma's DALL·E Mega. It has been stripped down for inference and converted to PyTorch. The only third-party dependencies are NumPy, requests, pillow, and torch". Now you can easily generate images using min-dalle on your machine (but it might take a while),

https://github.com/kuprel/min-dalle

Bonus - https://openai.com/blog/dall-e-2-pre-training-mitigations/

4 Things I Learned From Analyzing Menopause Apps Reviews - Dalya Gartzman, She Knows Health CEO, writes about 4 lessons she learned from analyzing Menopause Apps Reviews. I think it is interesting in 2 ways - app reviews are first, as a product-market fit strategy, to see what users are telling, asking, or complaining about in related.

https://medium.com/sheknows-health/4-things-i-learned-from-analyzing-menopause-apps-reviews-2cabf9ca9226

Inconsistent thoughts on database consistency - this post discusses the many aspects and definitions of consistency and how it is used in different contexts. I absolutely love those topics. Having said that, I wonder if people hold those discussions in real life and not just use common cloud-managed solutions encapsulating some of those concerns.

https://www.alexdebrie.com/posts/database-consistency/

Playing with DALL·E mini

Fri, 24 Jun 2022 19:40:18 GMT

DALL·E 2 is a multimodal AI system that generates images from text. OpenAI announced the model in April 2022. OpenAI is known for GPT-3, an autoregressive language model with 175 billion parameters. DALL·E 2 uses a smaller version of GPT-3. Read more here, here, and here (the last one also slightly discusses Google's image).

While the results look impressive at first sight, there are some caveats and limitations, including word order and compositionality issues, e.g., "A yellow book and a red vase" from "A red book and a yellow vase" are indistinguishable. Moreover, as one can see in the "A yellow book and a red vase" example below the images or more of the same, another drawback is that the system cannot handle negation, e.g., "A room without an elephant" will create, well, see below. Read more here.

Since I don't have access to DALL·E 2, I used DALL·E mini via Hugging Face for all the examples in this post. However, the two models experience the same issues.

A yellow book and a red vase

A room without an elephant

The model might have biases for example check all those software developers who write code, all men (also note that the face are very blurry in contrast to other surfaces in the images) -

software developer writing code

A CTO giving a talk

I decided to troll that a bit to find more limitations or point-out blind spots. Check out the following examples -

Object Oriented Programming

OOP

Object Disoriented Programming

Exploratory Data Analysis

EDA

The examples above demonstrate that model does not handle abbreviations well. I can think of several reasons for that, but that emphasizes the need to use precise wording and might need to try several times to get the desired result.

Trying negation again (in this case, the abbreviation worked okish) -

SQL

NoSQL

Structured Query Language

Which of course reminds all of us of this one -

And a few more -

SOLID principles

Clean Code

Computer Vision

To conclude, I cannot see a straightforward production-grade usage of this model (and it is anyhow not publically available yet) but maybe one use it for brainstorming and ideation. For me it feels like NLP in the days of TF-IDF there is yet a lot to come. Going forward I would love to have some more tunning possibilities like a color scheme or control the similarity between different results (mainly allow more diversity rather than more of the same).

5 interesting things (20/06/2022)

Mon, 20 Jun 2022 21:17:00 GMT

Visualizing multicollinearity in Python - I like the network one although it is not very intuitive at first sight. The others you can also get using pandas-profiling.

https://medium.com/@kenanekici/visualizing-multicollinearity-in-python-b5feedc9b3f1

Advanced Visualisations for Text Data Analysis - besides the suggested charts themselves, it is nice to get to know nxviz. I would actually like to see those charts as part of plotly as well.

https://towardsdatascience.com/advanced-visualisations-for-text-data-analysis-fc8add8796e2
Data Tales: Unlikely Plots - bar chart is boring (but informative :), but sometimes we need to think out of the box plot

https://medium.com/mlearning-ai/data-tales-unlikely-plots-1882c2a903da

XKCDs I send a lot - Is XKCD already an industry standard?

https://medium.com/codex/xkcds-i-send-at-least-once-a-month-1f6e9f9b6b89

5 Tier Problem Hierarchy - I use this framework to think of tickets I write, what is the expected input, output, and complexity, what I expect from each of my team members, etc.

https://typeshare.co/kimsiasim/posts/5-tier-problem-hierarchy-4718

Prioritize your Priority Score

Fri, 17 Jun 2022 09:23:45 GMT

A while ago, a friend asked me about a topic she needed to tackle – her team had many support tickets to prioritize, decide what to work on, and further communicate it to the relevant stakeholders.

They started as everyone starts – tier 1 and tier 2 support teams in their company stated the issue severity (low, medium, high) in the ticket, and they prioritized accordingly.

But that was not good enough. It was not always clear how to set the severity level – was it the client size or lifecycle stage, the feature importance, or anything else. Additionally, it was not granular enough to decide what to work on first.

We brainstormed, and she told me two important things for her: feature importance and client size. Both can be reduced to “t-shirt” size estimation, i.e., small client, medium client, large client, and extra-large client, and features of low/medium/high/crucial importance. Super, we can now generalize the single dimension axis system we previously had to two dimensions.

The priority score is now - $latex \sqrt{x^2+y^2}$

That worked great until they had a few tickets that got the same priority score, and they needed to decide what to work on and explain it outside of their team. The main difference between those tickets was the time it would take to fix each one. One would take several hours, one would take 1-2 days, and the last one would take two weeks and has high uncertainty. No problem, I told her – let’s add another axis – the expected time to fix. Time to fix can also be binned – up to 1 day, up to 1 week, up to 1 sprint (2 weeks), and longer. Be cautious here; the ax order is inverted – the longer it takes, the lower priority we want to give it.

The priority score is now - $latex \sqrt[\leftroot{-2}\uproot{2}3]{x_1^3+x_2^3+x_3^3}$

Then, when I felt we were finally there, she came and said – remember the time to fix dimension? Well, it is not as important as the client size and the feature importance. Is there anything we can do about it?

Sure I said, let’s add weights. The higher the weight is, the more influential the feature is. To keep things simple in our example, let’s reduce the importance of the time to fix compared to the other dimensions - $latex \sqrt[\leftroot{-2}\uproot{2}3]{x_1^3+x_2^3+0.5 x_3^3}$

To wrap things up

This score can be generalized to include as many dimensions as one would like - $latex \sqrt[\leftroot{-2}\uproot{2}n]{\sum_{i=1}^n w_i x_i^n}$.
I recommend keeping the score as simple and minimal as possible since it is easier to explain and communicate.
Math is fun and we can use relatively simple concepts to obtain meaningful results.

CSV to radar plot

Tue, 31 May 2022 19:58:26 GMT

I find a radar plot a helpful tool for visual comparison between items when there are multiple axes. It helps me sort out my thoughts. Therefore I created a small script that helps me turn CSV to a radar plot. See the gist below, and read more about the usage of radar plots here.

So how does it works? you provide a csv file where the columns are the different properties and each record (i.e line) is a different item you want to create a scatter for.

The following figure was obtained based on this csv -

https://gist.github.com/tomron/e5069b63411319cdf5955f530209524a#file-examples-csv

The data in the file is based on - https://www.kaggle.com/datasets/shivamb/company-acquisitions-7-top-companies

And I used the following command -

python csv_to_radar.py examples.csv --fill toself --show_legend --title "Merger and Acquisitions by Tech Companies" --output_file merger.jpeg

https://gist.github.com/tomron/e5069b63411319cdf5955f530209524a

Acing the Code Assignment Interview – Tips for Interviewers and Candidates

Fri, 29 Apr 2022 13:36:55 GMT

One of the most common practices today as part of the interview process are take-home assignments. However, though practical and valuable, this practice is tricky and needs to be used wisely to be beneficial. On the candidate's side, it is not enough to only solve the tasks, as there are a few more things you can do to make your submission shine and impress the reviewers. On the employer's side, companies have the challenge of creating a good assignment that will help assess the candidates and make the company attractive in the eyes of the candidate.

This week I took part in DevDays Europe Conference!
I was honored to participate in 2 sessions:

I moderated a "Leadership for Engineering Teams in Remote Work Era" session. The recording can be found here.

The session ended up with a few reading recommendations - Radical Candor, Six Simple Rules, Conscious Business, and The Promises of Giants.

I also talked about "Acing the Code Assignment Interview - Tips for Interviewers and Candidates", sharing my experience from our recruitment process at Lynx.MD. The recording can be found here and the slides are here.

A summary of my tips -

For candidates - plan ahead, document your work, and polish it by proofreading and linting just before handing it over, use version control tools and write tests to emphasize the added value you bring.

For companies - make the test relevant for the position and the candidate, respect the candidate's time and be available for her. Know your biases both when giving the assignment and when evaluating and giving feedback.

5 interesting things (18/02/2022)

Fri, 18 Feb 2022 15:49:54 GMT

What to Do When You Are Less Productive Than Your Teammates? I know Miri for a while and she has a very unique and sensitive point of view. This post is worth reading even if you don't share this feeling. It has some advice I find practical and it can help you better understand your colleagues and friends that might feel this way.

https://medium.com/@miryeh/what-to-do-when-you-are-less-productive-than-your-teammates-c5369423de8f

Wordle — What Is Statistically the Best Word to Start the Game With? Wordle conquered the world in the last few months therefore there must be a data science aspect to it.

https://medium.com/@noa.kel/wordle-what-is-statistically-the-best-word-to-start-the-game-with-a05e6a330c13

Bonus - https://mobile.twitter.com/bertiearbon/status/1484948347890847744

How I Discovered Thousands of Open Databases on AWS - In the last few months I have been training my security muscle to be more security aware both from infrastructure and code perspective and this is a great reminder why.

https://infosecwriteups.com/how-i-discovered-thousands-of-open-databases-on-aws-764729aa7f32

Top 10 Tips You Should Know As A Modern Software Architect - lately I tried to avoid such posts because I find

https://ankurkumarz.medium.com/top-10-tips-you-should-know-as-a-modern-software-architect-8e602c6c998f

Optimizing Workspace for Productivity, Focus, & Creativity - I think one of the things covid19 enabled us is to better question and adjust our workspace to our needs. This post shares some research, advice, and tips about the topic. The low ceiling vs high ceiling hooked me and I'm going to use those effects to better navigate discussions. After years of talking about it, I ordered a standup desk last week and I'm eager for it to arrive.

https://medium.com/@juanpabloaranovich/optimizing-workspace-for-productivity-focus-creativity-fcc0f28b6fa9

5 interesting things (29/12/2021)

Wed, 29 Dec 2021 21:57:49 GMT

7 PyTest Features and Plugins That Will Save You Tons of Time- I read many tutorials and posts about PyTest and this is the first time I run into those flags (features 1-5) which I find very useful. As always - if you can use your superpowers to read the documentation directly.

https://betterprogramming.pub/7-pytest-features-and-plugins-that-will-save-you-tons-of-time-74808b9d4756

Patterns for Authorization in Microservices - I find this post interesting since I currently face a similar problem of setting authorization and authentication architecture in the product I work on that can have complex access patterns such as a user can access multiple resources on different access levels owned by different organizations.

https://www.osohq.com/post/microservices-authorization-patterns

Database Indexing Anti-Patterns - I find this post slightly too high level. Yes, it state possible issues with indexing but a more effective post would be how to detect those anti-patterns on specific databases. E.g measure Mongo index usage on Mongo - https://docs.mongodb.com/manual/tutorial/measure-index-use/

This link is part of this post as a periodic reminder to think and take care of those topics before they become issues.
https://towardsdatascience.com/database-indexing-anti-patterns-dccb1b8ecef

How to safely think in systems. - "Effective systems thinking comes from the tension between model and reality, without a healthy balance you’ll always lose the plot.". I'm not sure if this post should be in the parenting category or in the career \ professional \ management category.
https://lethain.com/how-to-safely-think-in-systems/

How Improvised Stand-up Comedy Taught Me to Interview Better - "After all, questions in an interview are mostly a means for getting to know the candidate better, just as pulling words out of a hat is just the framework for a show.". Great post that connects two domains that usually aren't brought up together.

https://nogamann.medium.com/how-improvised-stand-up-comedy-taught-me-to-interview-better-9f0168be0726

We raised 12M$ seed funding

Fri, 24 Dec 2021 11:24:31 GMT

In the last year and 3 months, I work for Lynx.md. We develop a medical data science platform that bridges the gap between data owners and data consumers while taking care of de-identifications, privacy, and security aspects of sharing data.

10 days ago we announced that we raised a 12M$ seed funding and we are hiring - DevOps engineers, data engineers, data scientists, backend \ full-stack \ frontend developers, product managers. Our tech stack includes - Python mainly using FastAPI, Django, pandas, etc., AWS (but will soon add Azure too), Postgres, elastic search, Redis, Docker. Super interesting challenges with added value. Feel free to reach out if you want to learn more.

5 interesting things (1/12/21)

Wed, 01 Dec 2021 21:49:29 GMT

Tests aren’t enough: Case study after adding type hints to urllib3 - I read those posts as thrillers (and some of them are the same length :). This post describes the effort of adding type hints to urllib3 and what the maintainers' team learned during this process. Super interesting.

https://sethmlarson.dev/blog/2021-10-18/tests-arent-enough-case-study-after-adding-types-to-urllib3

Why you shouldn't invoke setup.py directly - post by Paul Ganssle, python core developer setuptool. TL;DR of the TL;DR in the post - "The setuptools team no longer wants to be in the business of providing a command-line interface and is actively working to become just a library for building packages". See the table in the summary section for a quick how-to guide.

https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html

Python pathlib Cookbook: 57+ Examples to Master It (2021) - I had a post about pathlib for a while in my drafts and now I can delete it since this guide is much more extensive. In short, pathlib is part of the Python standard library since Python 3.4 and it provides an abstraction for filesystem paths over different operating systems. If you still work with os for paths this is a good time to switch.

https://miguendes.me/python-pathlib

10 Tips For Using OKRs Effectively - I think a lot about OKRs for my team and moreover on personal OKRs and how to grow both the team and the product. I found this post (and the associated links) insightful.

https://rushabhdoshi.com/posts/2020-06-18-10-tips-for-making-okrs-effective/

How to Choose the Right Structure for Your Data Team - I started with those posts and soon enough read many more posts by y Bar Moses, Co-Founder, and CEO, Monte Carlo. Her posts have two dimensions that are relevant for me - team building (specifically around data-intensive products) and data engineering. If you find at least one of those topics interesting I believe you'll enjoy her posts.

https://towardsdatascience.com/how-to-choose-the-right-structure-for-your-data-team-be6c1b66a067
https://towardsdatascience.com/7-questions-to-ask-when-building-your-data-team-at-a-hypergrowth-company-dce0c0f343b4

Other pie chart

Fri, 22 Oct 2021 09:56:56 GMT

This morning I read "20 ideas for better data visualization". I liked it very much and specially I found 8th idea - "Limit the number of slices displayed in a pie chart" very relevant for me. So I jumped into the plotly express code and created a figure of type other_pie which given a number (n) and a label (other_label) created a pie chart with n sectors. n-1 of those sectors are the top values according to the `values` column and the other section is the sum of the other rows.

A gist of the code can be found here (check here how to build plotly)

I used the following code to generate standard pie chart and pie chart with 5 sectors -

import plotly.express as px
df = px.data.gapminder().query("year == 2007").query("continent == 'Europe'")
df.loc&#091;df&#091;'pop'] < 2.e6, 'country'] = 'Other countries' # Represent only large countries
pie_fig = px.pie(df, values='pop', names='country', title='Population of European continent')
otherpie_fig = px.other_pie(df, values='pop', names='country', title='Population of European continent', n=5, other_label="others")

And this is how it looks like -

Pie chart

Other pie chart

5 interesting things (21/10/21)

Thu, 21 Oct 2021 09:33:20 GMT

4 Things Tutorials Don't Tell You About PyPI - this hands-on experience together with the explanations is priceless. Even if you don't plan to upload a package to PyPI anytime soon those glimpses of how PyPI works are interesting.

https://blog.paoloamoroso.com/2021/09/4-things-tutorials-dont-tell-you-about.html

Responsible Tech Playbook - I totally agree with Martin Fowler statement that "Whether asked to or not, we have a duty to ensure our systems don’t degrade our society". This post promotes the text book about Responsible Tech published by Fowler and his colleagues from Thoughtworks. It also references additional resources such as Tarot Cards of Tech Ethical Explorer.

https://martinfowler.com/articles/2021-responsible-tech-playbook.html

A Perfect Match - A Python 3.10 Brain Teaser - Python 3.10 was released earlier this month and the most talked about feature is Pattern Matching. Read this post to make sure you get it correctly.

https://medium.com/pragmatic-programmers/a-perfect-match-ef552dd1c1b1

How I got my career back on track - careers is not a miracle. That's totally ok if you don't want to have one but if you do and have aspirations you have to own it and manage your way there.

https://rinaarts.com/how-i-got-my-career-back-on-track

PyCatFlow - A big part of current data is time series data combined with categorical data. E.g., change in the mix of medical diagnosis \ shopping categories over time etc. PyCatFlow is a visualization tool which allows the representation of temporal developments, based on categorical data. Check their Jupyter Notebook with interactive widgets that can be run online.

https://medium.com/@bumatic/pycatflow-visualizing-categorical-data-over-time-b344102bcce2

Growing A Python Developer (2021)

Fri, 15 Oct 2021 10:00:00 GMT

I recently run into a team lead question regarding how to grow a backend Python Developer in her team. Since I also iterated around this topic with my team I already had few ideas in mind.

Few disclaimers before we start. First, I believe that the developer also has a share in the process and should express her interest and aspirations. The team lead or tech lead can direct and light blind spots but does not hold all the responsibility. It is also ok to dive into an idea are a tool that is not required at the moment. They might come in handy in the future and they can inspire you. Second, my view is limited to the areas I work in. Different organizations or products have different needs and focus. Third, build habits to constantly learn and grow - read blogs and books, listen to podcasts, take online or offline courses, watch videos, whatever works for you as long as you keep moving.

Consider the links below as appetizers. Each subject below has many additional resources besides the ones that I posted. Most likely I'm just not familiar with them, please feel free to add them and I'll update the post. Some subjects are so broad and product dependent, e.g. cloud so I didn't add links at all. Additionally, when using a specific product \ service \ package read the documentation and make it your superpower. Know Python standard library well (e.g itertools, functools, collections, pathlib, etc), it can save you a lot of time, effort, and bugs.

General ideas and concepts

Clean code - book, book summary
Design patterns - refactoring book, refactoring guru, python design patterns GitHub repo
Distributed design patterns - Patterns of Distributed Systems
SOLID principles - SOLID coding in Python
Cloud
Deployment - CI\CD, docker, Kubernetes
Version control - git guide
Databases - Using Databases with Python, databases tutorials
Secure Development - Python cheat sheet by Snyk, OWASP

Python specific

Webservices - flask, Django, FastAPI
Testing - Unit Testing in Python — The Basics
Packaging - Python Packaging User Guide,
Data analysis - pandas, NumPy, sci-kit-learn
Visualization - plotly, matlpotlib
Concurrency - Speed Up Your Python Program With Concurrency
Debugging - debugging with PDB, Python debugging in VS Code
Dependency management - Comparison of Pip, Pipenv and Poetry dependency management tools
Type annotation - Type Annotations in Python
Python 3.10 - What’s New in Python 3.10?, Why you can’t switch to Python 3.10 just yet

Additional resources

Podcast.__init__ - The weekly podcast about Python and its use in machine learning and data science.
The real python podcast
Top 8 Python Podcasts You Should Be Listening to
Python 3 module of the week
Lazy programmer - courses on Udemy mainly AI and ML using Python
cloudonaut - podcast and blog about AWS

5 tips to ace coding interview assignments

Thu, 14 Oct 2021 08:39:36 GMT

Now days, it is a very common practice to give a coding home tests as part of interview process. Beside solving the task you are ask to I believe there are few additional things you can do in order to impress the reviewers and ace this step of the process.

1. Push the code to a private repository and share it with the reviewers - this creates a two fold advantage. First, it demonstrates the reviewers that you are familiar with version control tools and second it shows your working process and that you keep track of your work. Don't forget to write meaningful commit messages.

2. Write README file - the readme file gives a context to the entire project and reflects the way you understand the assignment. One of the annoting things as a reviewer is to guess how to run the code, what are the requirments and so on. Beside packaging or building the code in a way the runs smoothly (e.g in Python if using pip add a requirements.txt) a README file should help me find my way inside the project. In such assignments where you don't have a direct communication with the reviewers the README files also serves as a place to document your decisions and thoughts.

What should you include in the README file? Short introduction explaining the project purpose and scre. How to install and run or use it, preferably with some snippet that one can just copy-paste. How to run the tests (see next section :). Additional sections can include explainations about choices you made architecture wise or implementation wise, charts, performance evaluation, future ideas, dependencies, etc. This will help the reviewers get into your code quickly and run, understand your thinking and show that you are eager to share your knowledge with your peers.

For ease of use, check the template suggested here.

3. Write tests - usually unit tests are enough for this scope. This will help you debug your code and make sure it works properly. It will also signal the reviewers that you care about the quality of your code and know your job.

4. Run linters, spell check and do a proof reading for everything - make sure your code uses the common conventions and style for the tools you are using (e.g PEP-8 for python). Bonus points if you add the linters as pre-commit hooks to your repository. This make your code smoother and easier for the reviewers to read. The formatted code indicates that you are used to sharing your code with others and the hooks signal that you are productive and lazy by automating stuff.

5. Document everything - the idea behind this tip is not to annoy the reviewers by letting them guess what you meant. That is document what each module, function and parameter does. For example, in Python, use type annotation and docstrings.

5 interesting things (22/09/21)

Wed, 22 Sep 2021 12:09:22 GMT

Writing a Great CV for Your First Technical Role - a series of 3 parts about best practices, mistakes, and pitfalls in CV showing both good and bad examples. I find the posts relevant not just for first rolls but also as a good reminder when updating your CV.

https://naomikriger.medium.com/writing-a-great-cv-for-your-first-technical-role-part-1-75ffc372e54e

Patterns in confusing explanations - writing and technical writing are superpowers. Being able to communicate your ideas in a clear way that others can engage with is a very impactful skill. In this post, Julia Evans describes 13 patterns of bad explanation and accompanies that with positive examples.

https://jvns.ca/blog/confusing-explanations/

How We Design Our APIs at Slack - not only that I agree with those advices and had some bad experiences with similar issues both as a API supplier and consumer. Many times when big companies describe their architecture and process they are irrelevant to small companies due to cost, lack of data or resources or other reasons ,but the great thing about this post is that it also fits small companies and relatively easy to implement.

https://slack.engineering/how-we-design-our-apis-at-slack/

Python Anti-Pattern - this post describes a bug that is at the intersection of Python and AWS lambda functions. One can say that it is an extreme case but I tend to think it is more common than one would think and may spend hours debugging it. It is well written and very important to know if you are using lambda functions.

https://valinsky.me/articles/python-anti-pattern/

Architectural Decision Records - sharing knowledge is hard. Sometimes what is clear for you is not clear for others, sometimes it is not taken into account in the time estimation or takes longer than expected, other times you just want to move on and deliver. Having templates and conventions make it easier both for the writers and the readers. ADRs answer specific need.

https://adr.github.io/

5 interesting things (19/08/2021)

Thu, 19 Aug 2021 10:39:45 GMT

The 7 Branches of Software Gardening - "A small refactoring a day keeps the tech debt away " (paraphrasing^2). Great examples of small activities and improvments every developer can make on a daily basis and would pile up to big impact.

https://martinthoma.medium.com/the-6-branches-of-software-gardening-a90b3c0d6220

What is the right level of specialization? For data teams and anyone else - I like Erik Bernhardsson's posts, I linked to them more than once in the past. Bernhardsson highlights the tensions between being very professional and specific ("I only ETL process on Wednesdays 15:03-16:42 on Windowss Machines") versus being less proficient in more concepts and technologies . And this leads us to the next item.

https://erikbern.com/2021/07/23/what-is-the-right-level-of-specialization.html

7 Key Roles and Responsibilities in Enterprise MLOps - in this post Domino Data Lab introduce their view about the different roles and responsibilities in MLOps \ Data teams. For sure it is more suitable in enterprises, small and even medium companies cannot afford themselves and sometimes don't need all those roles and as suggested in Erik Bernhardsson's post, very specific specializaion make it harder to move people between teams according to the organization needs. Having said that, title is a signal (inside and outside the organizaiton) of what a person likes to do and which capabilities (no necessarily specific tools) s\he are probable to have.

https://blog.dominodatalab.com/7-roles-in-mlops/

The Limitations of Chaos Engineering - I decided to read a bit about Chaos engineering as I never experienced with it before and came across this post which is almost 4 years old. While it is important to validate the reselience of our architecture and its' implementation de facto, the common practice of fault injection also has its limitations which is good to know.

https://sharpend.io/the-limitations-of-chaos-engineering/

It's Time to Retire the CSV - if you ever worked with CSV you probably see this title and yell "Hell Yes!". If you want to gain an historic view and few more arguments have a look here -

https://www.bitsondisk.com/writing/2021/retire-the-csv/

pandas read_csv and missing values

Sun, 15 Aug 2021 08:58:03 GMT

I read Domino Lab post about "Data Exploration with Pandas Profiler and D-Tale" where they load diagnostic mammograms used in the diagnostic of breast cancer from UCI website. Instead of missiing values the data contains ?. When reading the data using pandas read_csv function naively interpret the value as string value and change the column type to be object instead of float in this case.

In the post mentioned above the authors dealt with the issue in the following way -

masses = masses.replace('?', np.NAN)
masses.loc&#091;:,names&#091;:-1]] = masses.loc&#091;:,names&#091;:-1]].apply(pd.to_numeric)

That is, they first replaced the ? values in all the columns with np.NAN and then convert all the columns to numeric. Let's call this method the manual method.

If we know the know the non default missing values in advance, can we do something better? The answer is yes!

See code here

Use na_values parameter

df = pd.read_csv(url, names=names, na_values=&#091;"?"])

na_values parameter can get scalar, string, list-like or dict parameters. If you pass a scalar, string or list-like parameter all columns are treated the same way. If you pass dict you can specify different set of NaN values per column.

The advantage of this method over the manual method is that you don't need to convert the columns after replacing the nan values. In the manual method the column types are specified (in the given case they are all numeric), if there are multiple columns types you need to know it and specify it in advance.

Side note - likewise, for non trivial boolean values you can use true_values and false_values parameters.

Use converters parameter

df = pd.read_csv(url, names=names, converters={"BI-RADS": lambda x: x if x!="?" else np.NAN})

This is usually used to convert values of specific columns. If you would like to convert values in all the columns in the same way this is not the preferred method since you will have to add an entry for each column and if new column is added you won't take care of it by default (this can be both advantage and disadvantage). However, for other use-cases, converters can help with more complex conversions.

Note that the result here is different then the result in the other methods since we only converted the values in one column.

Conclusion

Pandas provides several ways to deal with non-trivial missing values. If you know the non-trivial value in advance you are good to go and na_values is most likely the best way to go.

Performance wise (time) all methods perform roughly the same for the given dataset but that can change as a function on the dataset size (columns and rows), row types, number of non-trivial missing values.

On top of it, make reading documentation your superpower. It can use your tools smarter and more efficient and it can save you a lot of time.

See pandas read_csv documentation here

Things I learned today (04/08/2021)

Wed, 04 Aug 2021 12:39:01 GMT

AWS Lambda functions can now mount an Amazon Elastic File System (Amazon EFS)
AWS announcement

What is AWS Lambda?

AWS Lambda is FaaS (function as a service) offering. It is an event-driven, serverless computing platform which integrates with many other AWS services. For example you can trigger lambda function from API gateway, S3 event notification, etc.

AWS Lambda runtime includes Python, Node.js. ruby, Java, Go and C#.

It is very useful and cost-effective when you have infrequent and relatively short executions so you don't need to provision any infrastructure. Lambda has it's limitations, mainly it's running time - max 15 minutes. Storage was also a limitation up to this announcement but this is breakthrough.

What is Amazon EFS?

Amazon Elastic File System (EFS) is a cost-optimized file storage (not setup costs, just pay as you use) that can automatically scale from gigabytes to petabytes of data without needing to provision storage. It also allow multiple instances to connect to it simultaneously.

EFS are accessible from EC2 instances, ECS containers, EKS and AWS Fargate and AWS lambda.

Comparing to EBS, EFS is usually more expensive. However, the use case is different. EFS is a NFS file system (which means that it is not supported on Windows instances) and EBS is block storage and is usually not multi-attached (there are some EC2 + EBS configurations which allow multi-attach but that's not the main use case).

Why does it matter?

By default, lambda can /tmp storage of up to 512Mb this enables working with larger files. This means that you can import large machine learning models or packages. This also means that you can use an up-to-date version of files since it is easy to share.

Additional you can share information or state across invocations since EFS is a shared drive. I would not say it is optimal and generally I would rather to decouple it but it is possible and it is faster than S3.

In some cases it can also enable moving data intensive workloads (in AWS or on-premise) to AWS lambda and save cost.

See more here

Announcement - https://aws.amazon.com/blogs/aws/new-a-shared-file-system-for-your-lambda-functions/
Comparing Lambda data storage options - https://aws.amazon.com/blogs/compute/choosing-between-aws-lambda-data-storage-options-in-web-apps/
Lambda limitations - https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html

AWS Certified Solutions Architect – Associate

Wed, 04 Aug 2021 12:00:24 GMT

That's me :)

Took the exam yesterday and passed. I learned many news things on the way (e.g here, here, here and here) and I believe this knowledge (combined with the hands-on experience) would be very handful for me in the future.

Badge available here - https://www.credly.com/badges/8f7a4dec-70c7-407f-a2fc-775918f0cd64/public_url

Things I learned today (26/07/2021)

Mon, 26 Jul 2021 11:35:52 GMT

ElastiCache for Redis is HIPAA compliant while ElastiCache for Memcached is not

https://www.hipaaguide.net/wp-content/uploads/2018/05/hipaa-facts.png

What is ElastiCache?

ElastiCache is "Fully managed in-memory data store, compatible with Redis or Memcached. Power real-time applications with sub-millisecond latency" (here).

Most common use cases for ElastiCache are session store, general cache to increase throughput and decrease the load of other services or database, deployment of machine learning models and real time analytics.

AWS offers two flavours of ElastiCache - ElastiCache for Redis and ElastiCache for Memcached. To understand the difference better and recommendation on how to choose an engine see here.

What is HIPAA?

"The Healthcare Insurance Portability and Accountability Act (HIPAA) is an act of legislation passed in 1996 which originally had the objective of enabling workers to carry forward healthcare insurance and healthcare rights between jobs. "
https://www.hipaajournal.com/hipaa-explained/

Over the years and specifically after 2013 HIPAA rules were updated to fit to the technology development and expand the requirements to include business associates, where previously only covered entities were held to uphold the HIPAA restrictions.

Why does it matter?

Better safe than sorry - If you develop a product that needs to be HIPAA compliant it is better to choose in advance the right and compliant services rather than replacing it later

To read more -

AWS HIPAA Compliance - https://aws.amazon.com/compliance/hipaa-compliance/
AWS HIPAA Eligible Services - https://aws.amazon.com/compliance/hipaa-eligible-services-reference/
Difference between ElastiCache for Redis - https://aws.amazon.com/elasticache/redis-vs-memcached/
Choose an engine - https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/SelectEngine.html
ElastiCache compliance - https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/elasticache-compliance.html
HIPAA compliance for developers - https://github.com/truevault/hipaa-compliance-developers-guide

Things I learned today (23/07/2021)

Fri, 23 Jul 2021 10:04:45 GMT

S3 events notifications supports standard SNS topics and standard SQS queues as destinations but don't support SNS FIFO and SQS FIFO.

S3 events notifications enables you to be notified whenever a specific event happens in your bucket. To receive the notification you must define the events you are interested in and the destination. The notifications are usually triggered in seconds but sometimes can take longer.

The events are -

New object creation
Object removal (versioned and non-versioned objects)
Object restore (e.g from Glacier)
Object lost on a reduced redundancy storage
Object Replication

The possible destinations include -

SQS - as mentioned above standard queues only and not FIFO queues
SNS- as mentioned above standard topics only and not FIFO topics
Lambda

If when processing the events you write back to S3 be careful not to create an execution loop

See more here - https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html

Bonus for the weekend - bumped into this and I cannot deny (of service) that it can happen to me too

https://i.pinimg.com/originals/b1/46/c1/b146c11c3eb743ec2db7d40267ef2dab.jpg

Things I learned today (21/07/2021)

Wed, 21 Jul 2021 14:20:00 GMT

You can use Amazon SNS FIFO (first in, first out) topics and Amazon Simple Queue Service (Amazon SQS) FIFO queues together to provide strict message ordering and message deduplication
AWS documentation

While, SQS FIFO queues were introduced in 2016, SNS FIFO capabilities were introduced only on October 2020.

This capability is important for cases in which the order matters. E.g. bank transactions were you commit a transaction only if the balance remains non-negative.

Messages are grouped and ordered according to the message group ID. When sending a message you must specify a message group ID otherwise the action fails. If all the messages have the same message group ID then all the messages are sent and received in strict order. The message group id can be any value, e.g 12, "hello", "user_id-123", etc.

Note that as in the SQS case, the topic name must end with .fifo, a limitation that counts to the 80 characters restriction as well.

For further reading -

https://aws.amazon.com/blogs/aws/introducing-amazon-sns-fifo-first-in-first-out-pub-sub-messaging/
https://docs.aws.amazon.com/sns/latest/dg/sns-fifo-topics.html
https://aws.amazon.com/blogs/aws/new-for-amazon-simple-queue-service-fifo-queues-with-exactly-once-delivery-deduplication/

Things I learned today (20/07/2021)

Tue, 20 Jul 2021 09:36:00 GMT

Delay queues let you postpone the delivery of new messages to a queue for a number of seconds
AWS documentation

This means that all the messages which are pushed to this queue would be visible to the consumer after the delay period. The minimum delay which is also the default delay is 0 and the maximum is 15 minutes.

Note that when changing the delay of a queue the behaviour of FIFO queues and standard queues is different -

For standard queues, the per-queue delay setting is not retroactive—changing the setting doesn't affect the delay of messages already in the queue.
For FIFO queues, the per-queue delay setting is retroactive—changing the setting affects the delay of messages already in the queue.
AWS documentation

If you need to delay the visibility of specific messages and not all messages in the queue you can use message timers and add an initial invisibility period for a message. This is only supported by standard queues.Note that setting a message timer for individual messages overrides the delay period of the delay queue.

See the image below to understand message timeline in a queue -

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/images/sqs-delay-queues-diagram.png

See more here -
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-delay-queues.html
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-timers.html

Things I learned today (19/07/2021)

Mon, 19 Jul 2021 09:03:50 GMT

[AWS] independently map Availability Zones too names for each account
AWS documentation

This means that eu-west-1a in my account is not necessarily the same as eu-west-1a in your account.

Why does this matter? for example if you want to share subnets across accounts. Or maybe you want to ensure that services in different accounts are not in the same availability zone.

So how can you achieve this? use availability zone ids which are unique and consistent identifiers for availability zones.

See more here - https://docs.aws.amazon.com/ram/latest/userguide/working-with-az-ids.html

5 interesting things (02/07/2021)

Fri, 02 Jul 2021 14:28:00 GMT

Conducting a Successful Onboarding Plan and Onboarding Process - I believe that onboarding is important for the entire employment period. It helps setting expectations, getting to the code and being meaningful faster and assure both sides they made the right choice (and if not know it in an early stage). One thing I miss in this plan is the social part which I think is also important - having lunch \ coffee \ etc with not just the mentor.
I look forward to the next part "Conducting a Successful Offboarding Plan and Offboarding Process". It might sound like a joke, but it is not. Good offboarding process can help the organization learn and grow and leave the employee with a good taste so she might come back in the future or recommend her friends to join \ use the product.

https://blog.usejournal.com/conducting-a-successful-onboarding-plan-and-onboarding-process-6ec1b01ec2ae

The challenges of AWS Lambda in production - serverless is gaining popularity in the last years and specifically AWS lambda. While many times it sounds like a magic solution for scalability and isolation it also has its issues to know. In this post Lucas De Mitri from Sinch presents problems they run into and possible solutions. For a high level view on Lambda functions just read the conclusion part.

https://medium.com/wearesinch/the-challenges-of-aws-lambda-in-production-fc9f14b182be

My Arsenal Of AWS Security Tools - In a preview post I pointed out on ElectricEye a tool to continuously monitor your AWS services for configurations that can lead to degradation of confidentiality, integrity or availability. This github repo aggregates open source tools for AWS security: defensive, offensive, auditing, DFIR, etc.

https://github.com/toniblyx/my-arsenal-of-aws-security-tools

3 Problems to Stop Looking For in Code Reviews - I find the post title inaccurate but I like the attitude. As a reviewer you should not be bothered by tiny issues that can be enforced by tooling. Few tools are mentioned in the post and I would also add to that githooks which I find very powerful.I also agree with the insight that code reviews usually happen too late in the development process and constantly looking for the balance between letting developers progress and move forward and on the other hand give feedback on the right time.

https://medium.com/swlh/3-problems-to-stop-looking-for-in-code-reviews-981bb169ba8b

The Power of Product Thinking - In a previous post I mentioned that understanding the cost structure and trade-offs between different architecture (cost wise but also performance and feature wise) is a way to become a more valuable team member. Product thinking is another skill that can make you a more valuable and influential team member. This post explains what product thinking is (and isn't) and completes it by suggesting several practices on how to develop product thinking. Totally liked it and am going to adopt some of the suggested practices .

https://future.a16z.com/product-thinking/

AWS tagging best practices - 5 things to know

Wed, 30 Jun 2021 13:06:00 GMT

I read AWS tagging best practices whitepaper which was published in December 2018 and distilled 5 takeaways.

1. Use cases - tags have several use-cases including:

Cost allocation - using AWS Cost Explorer you can break down AWS costs by tag
Access Control - AM policies support tag-based conditions
Automation - for example tags can be used to opt into or out of automated task
AWS Console Organization and Resource Groups - e.g. create a custom console that organizes and consolidates AWS resources based on one or more tags
Security Risk Management - use tags to identify resources that require heightened security risk management practices
Operations Support - I find this use case tightly related to the automation use case

2. Standardized tag names and tag values -

There are only two hard things in Computer Science: cache invalidation and naming things.
Phil Karlton (check here)

A good practice as suggested in the whitepaper is to gather tagging requirements from all stakeholders and only then start implementing but a minimal step can be to define a convention for tags names and values that everyone can follow, see example from the document below.

tag names example

3. Cost allocation tags delay - this is something I experienced personally - "Cost allocation tags appear in your billing data only after you have (1) specified them in the Billing and Cost Management Console and (2) tagged resources with them". And even then it can take around 24 hours to appear, take it into account.

4. Tag everything - sounds trivial but sometimes organizations tag only some of the resources, tag everything you can to get a more comprehensive and accurate data of your expenses. A nice feature in the Billing and Cost Management Console is the ability to find resources the don't have a specific tags so you can easily find out what you missed.

5. Tags limitations - until 2016 AWS allowed up to 10 tags for a given resource. The current limit is 50. It definitely allows much more but it is still a limit to bear in mind when creating a tagging strategy. One way to avoid it is by using compound values, e.g. "anycompany:technical-contact = Susan Jones;sue.jones@anycompany.com; +12015551213" rather than a tag for each attribute (e.g. "anycompany:technical-contact-name = Susan Jones").

4 interesting things (24/06/2021) - Hebrew

Thu, 24 Jun 2021 11:37:00 GMT

5 years ago I published a blog post about 5 blogs I read in Hebrew. three of those are still live and kicking and I enjoy them (reversim, software archiblog, the bloggerit) the other two are no longer active. Additionally, in those 5 years podcasts became much more popular so I also included 2 podcasts I listen to.

Maya writes algorithms - Maya started this blog while studying for job interviews. In the first posts she presented questions she bumped into and her solutions (and the stream of thought that brought her to those solutions). Later posts also include writing about specific tools (e.g kubectl, git hooks) or interpersonal skills like talking in conferences, being prepared for a code review, etc.

https://algoritmim.co.il/

Internet Israel - Ran Bar Zik is a very experienced Full stack developer and Tech journalist who is also well known for his dad jokes. He writes mostly about Front End and security (but not only) and published several books about software development in Hebrew (those are usually very hard to find).

https://internet-israel.com/

Big Picture (podcast) - a podcast about tech and strategy. Each episode is a deep dive into the strategy of one company such as snapchat, spotify, twilio, etc.

https://bigpicture.buzzsprout.com/

No Tarbut (podcast)- podcast about the daily life of software engineering teams. The topics range from more tech related episodes such as monitoring, tools, etc. to so-called softer topics such as performance review, leading without authority, salary discussion, creating an inclusive culture.

http://notarbut.co/

5 interesting things (20/06/2021)

Sun, 20 Jun 2021 07:54:45 GMT

9 Steps to Software Project Handovers - handover is always a challenge and especially when a person leaves the organization and is no longer available for questions or able to access the resources. Similar issues can also arise when you leave the project for a while and then come back - you don't alway remember all the tricks you used to run the code or the tiny bits of each function. This post suggests practical steps and behaviours that many of them can be TL;DRed with the Zen of Python - "Explicit is better than implicit.".
https://betterprogramming.pub/9-steps-to-software-project-handovers-9325fbb72cfc

How to make an awesome Python package in 2021 - this post is a walk through building a python package. My main issue with it is the dependency management. I would put it in a different file, separate the development and usage dependency and lock the version - or shortly would rather use pipenv or poetry instead of pip but this could also be achieved with pip.

https://antonz.org/python-packaging/

A Simple Framework for Software Engineering Management - the suggested framework is indeed easy - 3 types of responsibilities (people management, delivery leadership and technical system ownership) vs 3 ranks of priorities (issues, things that are ok, ideas and aspirations). This framework is a good starting point for engineering leaders but also for engineers that can switch the people management with colleagues relations or similar or for personal growth.

https://medium.com/swlh/a-simple-framework-for-software-engineering-management-f70b216540f2

Full Cycle Data Science (FCDS) - this is a heavy read but worth it both for data practitioners and managers. Data science projects fail often. Sometimes it is because the problem is not defined well, other times because there is not enough data, data is not relevant, etc. FCDS tries to cast light and solve some of those problems - "In a nutshell, FCDS is a way of life that enables a single data practitioner to close the full product lifecycle and independently deliver end-to-end products, focusing only on where they bring added value".
https://towardsdatascience.com/fcds-b2d2e6b08d34

Endorse People Publicly, and Other Actions for Allies - Better Allies is an approach of making everyday actions to create inclusive and engaging workplaces. This post is a weekly newsletter where Karen Catlin (Advocate for Inclusive Workplaces) shares five simple actions to create a more inclusive workplace and be a better ally. This is a weekly reminder to be aware of biases and gaps and ideas to small and consistent changes that can make us and the people around us more comfortable and help everyone be the best version of themselves.
https://code.likeagirl.io/endorse-people-publicly-and-other-actions-for-allies-9352915c0956

5 interesting things - AWS edition (18/06/21)

Fri, 18 Jun 2021 09:37:01 GMT

As I collect items for my posts and wait until I have time to write about them I noticed I have many items related to AWS and decided to have a special edition.

12 Common Misconceptions about DynamoDB - many times our beliefs about certain tools or technology are based on hearing more than doing or doing but not getting into the depth of things and when running into a problem solving it with a solution we already know. This post describes features and qualities of DynamoDB that are sometimes ignored.

https://dynobase.dev/dynamodb-11-common-misconceptions/

Related Bonus - I really liked the link to Alex DeBrie post about single table design with DynamoDB

https://www.alexdebrie.com/posts/dynamodb-single-table/

AWS Chalice - it is not an official offering but rather a python code package for writing serverless applications. The syntax is very similar to Flask while there is a native support for local testing, AWS SAM and Terraform integration, etc. Disclaimer - if you are on multi-cloud I would not move from Flask or FastAPI to Chalice. Also note the used services (AWS lambda, AWS API Gateway, etc.) limits and make sure they don't limit your app.

https://aws.github.io/chalice/index

Related Bonus - auth0 tutorial on How to Create CRUD REST API with AWS Chalice
https://auth0.com/blog/how-to-create-crud-rest-api-with-aws-chalice/

ElectricEye - "ElectricEye is a set of Python scripts (affectionately called Auditors) that continuously monitor your AWS infrastructure looking for configurations related to confidentiality, integrity and availability that do not align with AWS best practices.". It is hard to know and follow all AWS best practices and this bundle of scripts is supposed to help uncover those. I have not tried it myself yet but it seems promising.
https://github.com/jonrau1/ElectricEye

My Comprehensive Guide to AWS Cost Control - computing and cloud costs take a big portion of every tech organization those days. Being a more valuable team member also means being aware of the costs and choosing wisely between the different alternatives.

https://corey.tech/aws-cost/

The Best Way To Browse 6K+ Quality AWS GitHub Repositories - most of the time we are not inventing the wheel and someone probably already did something very similar to what we are doing. Let's browse github to find it and accelerate our process.

https://app.polymersearch.com/discover/aws

Bonus - AWS snowball - I found out that this service exists only this week and it blew my mind - https://aws.amazon.com/snowball/

5 interesting things (06/05/2021)

Thu, 06 May 2021 12:15:46 GMT

How Hashicorp works - Hashicorp develops open-source products that are widely used in the industry including Terraform, Vault, Consul, etc. "How HashiCorp Works" provides a glimpse of Hashicorp's culture and practices. I appreciate this kind of transparency and chance to learn.

https://works.hashicorp.com/

Make boring plans - a more accurate title would be "make predictable plans". That is, the next tasks should be predictable based on the team's knowledge regarding the product pains, bug, customers' requests, etc.A possible good way to measure how boring the plans are is to ask the team to prioritize the top-k tasks we should work on in the next period (quarter \ sprint, etc.) and check if the tasks overlap. Disclaimer - each team member has its' own view, pain points, and features they would like to develop and might be biased towards it.

https://skamille.medium.com/make-boring-plans-9438ce5cb053

Explainable AI Cheat Sheet - cheat sheet, video and resources regarding XAI. This is a very good way to get into this field.

https://ex.pegg.io/

I’ve code reviewed over 750 pull requests at Amazon. Here’s my exact thought process - code review is an art and is a way personal relations manifest themselves. One day I might write a longer post about code reviews but for now I want to focus on the last 2 points in this post - "I approve when the PR is good, not perfect" and "I seek feedback for whether I’m reviewing well". "Good not perfect" - this depends on the team standards, DoD, the PR scope, etc. Specifically, in startups when the time and money are limited each delay has its' costs. "I seek feedback" - how is the quality of my CR is measured? what are the goals of CR (familiarity with the code, finding bugs, enforcing standards, something else?)?. I would like to see or find ways to assess the quality of the CR and give feedback to the code reviewer.

https://curtiseinsmann.medium.com/ive-code-reviewed-over-750-pull-requests-at-amazon-here-s-my-exact-thought-process-cec7c942a3a4

My Clean and Tidy Checklist for Clean and Tidy Data - it is commonly believed that "Data scientists spend 80% of their time cleaning data". This post provides a conceptual framework to clean data so the time data scientist spend on cleaning data might drop to 79% ;-)

https://towardsdatascience.com/my-clean-and-tidy-checklist-for-clean-and-tidy-data-fbdeacb3736c

5 interesting things (23/04/21)

Fri, 23 Apr 2021 09:06:43 GMT

You Are Probably Not Making The Most of Pandas “read_csv” Function - this might seems trivial and everything can be found in the documentation but it is well served here with many examples.
https://towardsdatascience.com/you-are-probably-not-making-the-most-of-pandas-read-csv-function-51bcf069e646

Disasters I've seen in a microservices world - I experienced most of the disasters described in this post and totally agree with the bottom line - "These edge cases become the new normal at a certain scale, and we should cope with them."

https://world.hey.com/joaoqalves/disasters-i-ve-seen-in-a-microservices-world-a9137a51

Chess2Vec - while there are many x2Vec works in recent years this work is passion-based. The writer informatics profess that wanted to apply the algorithm to a hobby of his - chess. I think this is a great example of side project and I would love to see more such combinations.
https://towardsdatascience.com/chess2vec-map-of-chess-moves-712906da4de9

Driving Cultural Change Through Software Choices - there are several approaches on who to drive changes this post presents a somehow more immediate approach, straight-forward and role model approach. The author's idea is that if you choose or provide the tools that reflect your values your team will also adopt them.

https://skamille.medium.com/driving-cultural-change-through-software-choices-bf69d2db6539

Letter to (new) managers - an insightful post for managers and people who strive to become managers. Two quotes I liked - "Trust is consistency over time" and "We start managing others the way we manage ourselves, but to do better, we need to learn new tools and use them adaptively.". Managing others the way we manage ourselves is one of the most common mistakes I saw managers do and I try to be super aware to it myself.

https://productlessons.substack.com/p/letter-to-new-managers

5 tips for using Pandas

Mon, 14 Dec 2020 14:45:55 GMT

Recently, I worked closely with Pandas and found out a few things that are might common knowledge but were new to me and helped me write more efficient code in less time.

1. Don't drop the na

Count the number of unique values including Na values.

Consider the following pandas DataFrame -

df = pd.DataFrame({"userId": list(range(5))*2 +[1, 2, 3],
                   "purchaseId": range(13),
                   "discountCode": [1, None]*5 + [2, 2, 2]})

Result

If I want to count the discount codes by type I might use - df['discountCode'].value_counts() which yields -

1.0    5
2.0    3

This will miss the purchases without discount codes. If I also care about those, I should do -

df['discountCode'].value_counts(dropna=False)

which yields -

NaN    5
1.0    5
2.0    3

This is also relevant for nuniqiue. For example, if I want to count the number of unique discount codes a user used - df.groupby("userId").agg(count=("discountCode", lambda x: x.nunique(dropna=False)))

See more here - https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.nunique.html

2. Margin on Row \ columns only

Following the above example, assume you want to know for each discount code which users used it and for each user which discount code she used. Additionally you want to know has many unique discount codes each user used and how many unique users used each code, you can use pivot table with margins argument -

df.pivot_table(index="userId", columns="discountCode",
               aggfunc="nunique", fill_value=0,
               margins=True)

Result -

It would be nice to have the option to get margins only for rows or only for columns. The dropna option does not act as expected - the na values are taken into account in the aggregation function but not added as a column or an index in the resulted Dataframe.

3. plotly backend

Pandas plotting capabilities is nice but you can go one step further and use plotly very easy by setting plotly as pandas plotting backend. Just add the following line after importing pandas (no need to import plotly, you do need to install it) -

pd.options.plotting.backend = "plotly"

Note that plotly still don't support all pandas plotting options (e.g subplots, hexbins) but I believe it will improve in the future.

See more here - https://plotly.com/python/pandas-backend/

4. Categorical dtype and qcut

Categorical variables are common - e.g., gender, race, part of day, etc. They can be ordered (e.g part of day) or unordered (e.g gender). Using categorical data type one can validate data values better and compare them in case they are ordered (see user guide here). qcut allows us to customize binning for discrete and categorical data.

See documentation here and the post the caught my attention about it here - https://medium.com/datadriveninvestor/5-cool-advanced-pandas-techniques-for-data-scientists-c5a59ae0625d

5. tqdm integration

tqdm is a progress bar that wraps any Python iterable, you can also use to follow the progress of pandas apply functionality using progress_apply instead of apply (you need to initialize tqdm before by doing tqdm.pandas()).

See more here - https://github.com/tqdm/tqdm#pandas-integration

5 interesting things (04/12/2020)

Fri, 04 Dec 2020 12:03:46 GMT

How to set compensation using commonsense principles - yet another artwork by Erik Bernhardsson. I like his analytics approach and the way he models his ideas. His manifest regarding compensation systems (Good/bad compensation systems) is brilliant. I believe most of us agree with him while he put it into words. His modeling has some drawbacks that he is aware of. For example, assuming certainty in employee productivity, almost perfect knowledge of the market. Yet, it is totally worth your time.

https://erikbern.com/2020/06/08/how-to-set-compensation-using-commonsense-principles.html

7 Over Sampling techniques to handle Imbalanced Data - imbalanced data is a common real world scenario, specifically in healthcare where most of the patients don't have a certain condition one is looking for. Over-sampling is a method to handle imbalanced data, this post describes several techniques to handle it. Interestingly, at least in this specific example, most of the techniques do not bring significant improvement. I would therefore compare several techniques and won't just try one of them.
https://towardsdatascience.com/7-over-sampling-techniques-to-handle-imbalanced-data-ec51c8db349f

This post uses a the following package which I didn't know before (it would be great if it could become part of scikit-learn) - https://imbalanced-learn.readthedocs.io/en/stable/index.html

It would be nice to see a similar post fo downsampling techniques.

Python's do's and don't do - very nicely and written with good examples -
https://towardsdatascience.com/10-quick-and-clean-coding-hacks-in-python-1ccb16aa571b

Every Complex DataFrame Manipulation, Explained & Visualized Intuitively - can't remember how pandas function work? great, you are not alone. You can use this guide to quickly remind you how melt, explode, pivot and others work.
https://medium.com/analytics-vidhya/every-dataframe-manipulation-explained-visualized-intuitively-dbeea7a5529e

Causal Inference that’s not A/B Testing: Theory & Practical Guide - Causality is often overlooked in the industry. Many times you developed a model that is "good enough" and move on. However, this might increase bias and lead to unfavourable results. This post suggests a hands-on approach to causality accompanied by code samples.

https://towardsdatascience.com/causal-inference-thats-not-a-b-testing-theory-practical-guide-f3c824ac9ed2

Plotly back to back horizontal bar chart

Wed, 11 Nov 2020 20:44:57 GMT

Yesterday I read Boris Gorelik post - Before and after: Alternatives to a radar chart (spider chart) and I also wanted to used this visualization but using Plotly.

So I created the following gist -

https://gist.github.com/tomron/8b2f5872a28101445a9205d9e735099d

How to manage remote teams - 3 insights from GitLab's course in Coursera

Wed, 21 Oct 2020 11:12:15 GMT

I have recently listened to GitLab's "Remote management course" in Coursera. While there are companies that are remote-first and were built like that or chose to adopt this structure (or other remote work structures) with the outbreak of COVID-19 many companies were forced to change and adopt remote work. This course timing is relevant than ever. Here are few insights -

This course is a great PR for GitLab they present a well-studied background and ideas of remote work, including pros, cons, and trade-offs. They present their take on remote work and how it is implemented in GitLab and refer to their own handbook and resources. Additionally, course lecturers are very diverse, which I believe can be attractive to many candidates.
Physical health and specifically mental health are mentioned multiple times during the course. Working remotely raises mental health challenges which are important to mention and I'm glad they did it. Specifically during the pandemic where interactions with other decreases and many employees didn't choose this form of work in advance.
What is a vital capability for remote employees? communication. They preach to value strong communication skills and emphasize that it is crucial in an asynchronous work environment where your colleagues are in different time zones, have varied cultural backgrounds and it might take them hours or days to answer. Well, I think it is also important for colocated employees. Interactions with others shape our days, sometimes more than our actual tasks and it is important to be in an environment where we feel comfortable and even if we have disagreements (and I would be worried if there aren't) they can be discussed and settled. Generally, I find many of their ideas also relevant for colocated work.

Personally, I like working remotely as it saves the commute and allows me flexible working hours. However, in the past, I had the experience of being the only remote employee and that didn't work well. Many of the communication was in the coffee corner and was not accessible to me, promotion paths were blocked, etc.

If you are looking for a remote position, I strongly suggest this great resource - Established remote companies.

Getting started with Fairness in ML - 5 resources

Sun, 26 Jul 2020 10:14:19 GMT

Mirror Mirror - Reflections on Quantitative Fairness - this is one of the first pieces I read about algorithmic fairness and caught my attention. It surveys the most common definitions of fairness together with relevant examples in a readable format.

https://shiraamitchell.github.io/fairness/

Fairness in Machine Learning book - a WIP of an online textbook about fairness and machine learning by very notable researchers in this field - Solon Barocas, Moritz Hardt, Arvind Narayanan.

https://fairmlbook.org/

Equality of Opportunity in Supervised Learning - if you want to read a research paper about fairness this paper by Moritz Hardt, Eric Price and Nathan Srebro is a good starting point. It is central and relatively easy to read. It defines "Equalized odd" and "Equal opportunity" fairness measures which are commonly used and also gives a geometric intuition.

https://arxiv.org/abs/1610.02413

Responsible Data Science course - a wider view than just fairness. This course is given by Julia Stoyanovich at New York University and includes topics in data cleaning, anonymity explainability, etc. If you look for a pertinent reading list you can find it there.

https://dataresponsibly.github.io/courses/spring20/

AIF360 - A tooling containing metrics for datasets and models to test for biases and algorithms to mitigate bias in datasets and models. The package is available in both Python and R.
AIF360 was originally developed by IBM and was recently donated to Linux Foundation AIAmong the currently available software tools, I find this the most baked and stable one.

https://github.com/Trusted-AI/AIF360

5 interesting things (8/6/2020)

Mon, 08 Jun 2020 08:57:31 GMT

DBScan practitioners guide - DBScan is a density-based clustering method. One important advantage comparing to K-means is DBScan's ability to identify noise and outliers. I feel that DBScan is often under-estimated. See this guide to learn more on how DBScan works, how to choose hyperparameters and more.

https://towardsdatascience.com/a-practical-guide-to-dbscan-method-d4ec5ab2bc99

Fourier Transform for Data Science - When I was in undergrad school I learned FFT out of context, it was just an algorithm in the textbook and I didn't understand what it was good for. Later I was asked about it in an oral in grad school and was able to mumble something. Much later I tried to pull some analysis on ECG waves and then I finally understood what it was about.Read this post if you want to demystify Fourier transform.

https://medium.com/swlh/fourier-transformation-for-a-data-scientist-1f3731115097

Bonus - OpenCV tutorial on Fourier Transform

https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_transforms/py_fourier_transform/py_fourier_transform.html

Dataset shift - dataset shift happens when the test set and the train set come from different distributions. There are multiple expressions of this phenomenon, such as covariate shift, concept shift, prior distribution shift. I believe that every data scientist working in the industry came across at least one of those manifestations. This post provides a very good introduction to the topic and useful links if you want to delve.

https://towardsdatascience.com/understanding-dataset-shift-f2a5a262a766

A Practical Framework for AI Adoption: A Five-Step Process - having several years of experience as a data scientist I have noticed that data products are often not deployed, do not meet stakeholders' expectations, not used as the data scientist intended, etc. This post introduces a framework that tries to remedy some of those problems.

https://datascientia.blog/2020/05/23/framework-for-ai-adoption-a-five-step-process/

Diagrams as code - a code-based tool to draw system diagrams, possibly easier than fighting draw.io. It contains many icons including several cloud providers (AWS, GCP, Azure, etc). common servicers (K8S, Elastic, spark), etc. All in all, seems very promising.

https://diagrams.mingrammer.com/

Connected Papers is Here

Wed, 03 Jun 2020 11:57:00 GMT

Almost a year ago I published a blog post about "5 ways to follow publications in your field". Yesterday I was exposed to a new tool - Connected Papers.

Connected Papers present graph of papers that are similar to each other. However, note that this is not a citation tree. Each paper is a node and there is an edge between papers if they are similar. The similarity matrix is based on co-citation and co-references.
The node's colour represent the publication year and the node's size correspond to the number of citation. The graphs are built on the fly given a link or title.

The interface presents the papers' abstract which makes it easier to browse and jump between the different graphs.

Two small features I can think of is to filter papers according to a publication year and an option to download citation (i.e. bibtex, APA).

I believe that I'll used it extensively when working on related work.

5 interesting things (11/3/2020)

Wed, 11 Mar 2020 10:59:10 GMT

Never attribute to stupidity that which is adequately explained by opportunity cost - if you have a capacity for only 10 words, those are the 10 words you should take - "prioritization is the most value creating activity in any company". One general, I really like Erik Bernhardsson writing and ideas, I find his posts insightful.

https://erikbern.com/2020/03/10/never-attribute-to-stupidity-that-which-is-adequately-explained-by-opportunity-cost.html

Causal inference tutorials thread - check this thread if you are interested in causal inference.

https://twitter.com/juli_schuess/status/1177155098306318336

The (Real) 11 Reasons I Don't Hire you - I follow Charity Major's blog since I heard here in ״Distributed Matters" in 2015. Besides being a very talented engineer, she also writes about being a founder and manager. Hiring and is hard for both sides and although it is hard to believe it is not always personal and a candidate has to be a good match for the presented company, for the future company, to the team. The company would like to have exciting and challenging tasks for the candidate so she will be happy and grow in the company, And of course, we are all human and make mistakes from time to time. Read Charity's post in order to examine the not hiring decision from the employer's point of view.

https://charity.wtf/2019/10/18/the-real-11-reasons-i-dont-hire-you/

The 22 Most-Used Python Packages in the World - an analysis of the most downloaded Python packages on PyPI over the past 365 days. Some of the results are surprising - I expected pip to be the most used package and it is only the fourth after urllib3, six and boto core, and requests to be ranked a bit higher. Some of the packages are internals we are not even aware of such as idna and pyasn1. Interesting reflection.

https://medium.com/better-programming/the-22-most-used-python-packages-in-the-world-7020a904b2e

Time-based cross-validation - it might seem obvious when reading but there are few things to notice when training a model that has time dependency. This post also includes Python code to support the suggested solutions.

https://towardsdatascience.com/time-based-cross-validation-d259b13d42b8

DevSkiller 2019 report - few comments

Thu, 05 Mar 2020 20:29:01 GMT

I read a blog post about DevSkiller report analyzing some trends and I read the original report and I have some comments to make. “Companies from Israel are the most selective” According to the post, companies in Israel are considered the most selective since they consider only 12.26% of the developers they test. Another point of view is that Israeli companies give less weight to résumés and would rather test the applicant skills. This can increase the diversity and give a chance to more people. I think that it is a good practice for an industry that lacks more than 10,000 professionals (see here) “72% of companies are looking for JavaScript developers and JS is the most popular IT skill developers are tested in (40%)” JavaScript is used both front-end (e.g. react) and back-end tool (e.g. node.js) and I think that the popularity . I think grouping it together is too broad. Also, the SQL skill is somehow a side requirement of many positions and is almost worthless on its own. The big gap between JavaScript (40%) and HTML\CSS (20%) is weird, specifically when they compare it to the StackOverflow 2019 report where they were JavaScript had 67.8% and HTML\CSS have 63.5% and they were one after the other. The numbers themselves do not matter, just the gap and the order. On a side comment, interpersonal skills such as team player, leadership, responsibility are ignored in this report and that’s a shame. They are sometimes more important when hiring someone. “React, Spring, ASP.NET, MySQL, HTML, Data Analysis, and Laravel are the most popular technologies in their respective tech stacks” CSS is a tech stack? Weird analysis from my point of view, maybe web development would have been a better title. Is it a surprise that HTML and CSS are tested together? I find it hard to believe that an employer looks for an employee that is skilled only with one of those, they are tightly coupled. In this section in the post, I miss a mention of NoSQL technologies. General comments Obviously, they use their data which is great but there might be a selection bias to pay attention to. For example, companies that look for JavaScript developers are not able to screen for themselves and therefore use their services. Comparing to companies the loop for Python developers. In the section about technologies tested together, they point out that the most common combination last year was Java+SQL and this year it is JavaScript + CSS. Maybe their screening service for Java+SQL is not as good as their screening for JavaScript + CSS and therefore companies do not use it. The completion rate of the test is impressive (93%). There is a detailed analysis of the geography of the hiring companies and the candidates’ origins. I am also interested in demographics such as age, gender, marital status. I know that not everything is legal to have or ask. But I wonder if parents are more likely or less likely to complete the tests (or even start them). Are women as likely as men to pass the tests? Are there feminine and masculine technologies? Data about years of experience and the correlation to technologies used and the probability passing the tests would also be interesting.

How To Capitalize All Words in a String in Python - The Easy way

Tue, 03 Mar 2020 08:39:06 GMT

I run into this medium post today - "How To Capitalize All Words in a String in Python". It explains how to convert “hello world” to “Hello World” but it does it the hard way. An easier solution would be to use title function.

x = "hello world"
print(x.title())
#Hello World

See more documentation here

argparse choices

Sun, 23 Feb 2020 08:56:36 GMT

I saw this post in Medium regarding argparse, which suggests the following -

parser.add_argument("--model", help="choose model architecture from: vgg19 vgg16 alexnet", type=str)

I think the following variant is better -

parser.add_argument("--model", help="choose model architecture from: vgg19 vgg16 alexnet", type=str, choices=['vgg19', 'vgg16', 'alexnet'], default='alexnet'])

If an illegal parameter is given, for example --model vgg20, the desired behavior of almost every program is to throw an exception. This won't happen in the first case. If the user mistypes the model name the script will use Alexnet pre-trained instead of throwing an exception (implemented later in the script). Using the choices argument will solve this. Adding default='alexent', takes care in the case where the user does not choose a model actively. For the example presented in the original post this is the desired behavior. See more here -

Python documentation
Python module a week
argparse cookbook

Missing data in Python - 5 resources

Sun, 16 Feb 2020 11:47:22 GMT

Bonus - R-miss-tastic - theoretical background and resources which relate to R missing values package. I recommend the lecture notes. https://rmisstastic.netlify.com/lectures/ Working with missing data in Pandas - pandas is the swiss knife of data scientists, Pandas allows dropping records with missing values, fill missing values, interpolation of missing data points, etc. https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html Missing data visualization - provides several levels and types of visualizations - per sample, per feature, features heat map and dendrogram in order to gain a better understanding of missing values in a dataset. https://github.com/ResidentMario/missingno FancyImput - Multivariate imputation and matrix completion algorithms implemented. This package was partially merged to scikit-learn. This package focus on viewing the data as a matrix and not a composition of columns, unfortunately, it is no longer actively maintained but maybe in the future. https://github.com/iskandr/fancyimpute Missingpy - scikit-learn consistent API for data imputation. Implements KNN imputation (also implemented in FancyImput) and Random Forest imputation (MissForest). Seems unmaintained. https://github.com/epsilon-machine/missingpy MDI - Missing Data Imputation Package - accompanying code to Missing Data Imputation for Supervised Learning (https://arxiv.org/abs/1610.09075) https://github.com/rafaelvalle/MDI

3 follow-up notes on 3 python list comprehension tricks

Sun, 26 Jan 2020 10:00:27 GMT

I saw the following post about list comprehension tricks in Python. I really like python comprehension functionality - dict, set, list, I don’t discriminate. So 3 follow up notes about this post -

1. Set Comprehension

Beside dictionary and lists, comprehensions also work for sets - [sourcecode language="python" wraplines="false" collapse="false"] {s for s in [1, 2, 1, 0]} #set(0,1,2)) {s**2 for s in [1,2,1,0,-1]} #set(0,1,4) [/sourcecode]

2. Filtering (and a glimpse to generators)

In order to filter a list, one can iterate over the list or generator, apply the filter function and output a list or can use the build-in filter function and receive a generator that is more efficient as described further in the original post. [sourcecode language="python" wraplines="false" collapse="false"] words = ['deified', 'radar', 'guns'] palindromes = filter(lambda w: w==w[::-1], words) list(palindromes) #['deified', 'radar'] [/sourcecode] Additional nice to know the build-in function is the map function, that for example can yield the words’ lengths as generators - [sourcecode language="python" wraplines="false" collapse="false"] words = ['deified', 'radar', 'guns'] lengths = map(lambda w: len(w), words) list(lengths) #[7, 5, 4] [/sourcecode]

3. Generators

Another nice usage of generators is to create an infinite sequence - [sourcecode language="python" wraplines="false" collapse="false"] def infinite_sequence(): num=0 while True: yield num num+=1 gen = infinite_sequence() next(gen) #0 next(gen) #1 next(gen) #2 [/sourcecode] Generators can be piped, return multiple outputs, and more. I recommend this postto a better understand generators.

3 interesting features of NetworkX

Fri, 17 Jan 2020 20:54:59 GMT

“NetworkX is a Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks.” NetworkX lets the user create a graph and then study it. For example - find the shortest path between nodes, find node degree, find the maximal clique, find coloring of a graph and so on. In this post, I’ll present a few features I find interesting and are maybe less known. Multigraphs Multigraph is a graph that can store multiedges. Multiedges are multiple edges between two nodes (it is different from hypergraph where an edge can connect any number of nodes and no just two). NetworkX has 4 graph types - the well-known commonly used directed and undirected graph and 2 multigraphs - nx.MultiDiGraph for directed multigraph and nx.MultiGraph for undirected multigraph. In the example below, we see that if the graph type is not defined correctly, functionalities such as degree calculation may yield the wrong value - [sourcecode language="python" wraplines="false" collapse="false"] import networkx as nx G = nx.MultiGraph() G.add_nodes_from([1, 2, 3]) G.add_edges_from([(1, 2), (1, 3), (1, 2)]) print(G.degree()) #[(1, 3), (2, 2), (3, 1)] H = nx.Graph() H.add_nodes_from([1, 2, 3]) H.add_edges_from([(1, 2), (1, 3), (1, 2)]) print(H.degree()) #[(1, 2), (2, 1), (3, 1)] [/sourcecode] Create a graph from pandas dataframe Pandas is the swiss knife of every data scientist, so naturally, it would be a good idea to create a graph from pandas dataframe. The other way around is also possible. See the documentation here. The example below shows how to create a multigraph from a pandas dataframe where each edge has a weight property. [sourcecode language="python" wraplines="false" collapse="false"] import pandas as pd df = pd.DataFrame([[1, 1, 4], [2, 1, 5], [3, 2, 6], [1, 1, 3]], columns=['source', 'destination', 'weight']) print(df) # source destination weight # 0 1 1 4 # 1 2 1 5 # 2 3 2 6 # 3 1 1 3 G = nx.from_pandas_edgelist(df, 'source', 'destination', ['weight'], create_using=nx.MultiGraph) print(nx.info(G)) # Name: # Type: MultiGraph # Number of nodes: 3 # Number of edges: 4 # Average degree: 2.6667 [/sourcecode] Graph generators One of the features I find the most interesting and powerful. The graph generator interface allows creating several types we just one line of code. Some of the graphs are deterministic given a parameter (e.g complete graph of k nodes) while some are random (e.g. binomial graph). Below are a few examples of deterministic graphs and random graphs. The examples below are the tip of the iceberg of the graph generator capabilities. Complete graph - creates a graph with n nodes and an edge between every two nodes. Empty graph - creates a graph with n nodes and no edges. Star graph - create a graph with one central node connected to n external nodes. [sourcecode language="python" wraplines="false" collapse="false"] G = nx.complete_graph(n=9) print(len(G.edges()), len(G.nodes())) # 36 9 H = nx.complete_graph(n=9, create_using=nx.DiGraph) print(len(H.edges()), len(H.nodes())) # 72 9 J = nx.empty_graph(n=9) print(len(J.edges()), len(J.nodes())) # 0 9 K = nx.star_graph(n=9) print(len(K.edges()), len(K.nodes())) # 9 10 [/sourcecode] Binomial Graph - create a graph with n nodes and each edge is created with probability p (alias for gnp_random_graph and erdos_renyi_graph). [sourcecode language="python" wraplines="false" collapse="false"] G1 = nx.binomial_graph(n=9, p=0.5, seed=1) G2 = nx.binomial_graph(n=9, p=0.5, seed=1) G3 = nx.binomial_graph(n=9, p=0.5) print(G1.edges()==G2.edges(), G1.edges()==G3.edges()) # True False [/sourcecode] Random regular graph - creates a graph with n nodes, edges are created randomly and each node has degree d. [sourcecode language="python" wraplines="false" collapse="false"] G = nx.random_regular_graph(d=4, n=10) nx.draw(G) plt.show() [/sourcecode] Random tree - create a uniformly random tree of n nodes. [sourcecode language="python" wraplines="false" collapse="false"] G = nx.random_tree(n=10) nx.draw(G) plt.show() [/sourcecode] All the code in this post can be found here Additional Resource Official site SO questions https://www.datacamp.com/community/tutorials/networkx-python-graph-tutorial https://www.geeksforgeeks.org/directed-graphs-multigraphs-and-visualization-in-networkx/amp/

5 interesting things (23/09/2019)

Mon, 23 Sep 2019 10:35:42 GMT

All the best engineering advice I stole from non-technical people - the length of this post could definitely be cut into half. However, I find the idea that useful, meaningful advice and insights can pop around and you just need to listen importantly. There is something to learn from everyone, and we just need to be willing to do that.

https://medium.com/@bellmar/all-the-best-engineering-advice-i-stole-from-non-technical-people-eb7f90ca2f5f

The Ultimate Guide to Acing Your First Coding Interview - looking for a job and especially your first job as a junior developer is a hard job. Read Dana's tips here -

(full disclosure - I know Dana and helped to edit the post a bit)

https://medium.com/@dana.blanc/the-ultimate-guide-to-acing-your-first-coding-interview-5cd78748ca6d

Python Libraries for Interpretable Machine Learning - Interpretability in machine learning becomes an important topic lately. While I have my doubts about it and what interpretability really is, this post presents the main Python packages for interpretability -

https://towardsdatascience.com/python-libraries-for-interpretable-machine-learning-c476a08ed2c7

Bonus point - missing package SHAP(SHapley Additive exPlanations) - https://github.com/slundberg/shap

Document Embedding techniques - (another) excellent post by Shay Palachy which reviews the main and prominent approaches regarding document embedding including many references to the relevant literature.

https://towardsdatascience.com/document-embedding-techniques-fed3e7a6a25d

Style Pandas Dataframe Like a Master - while I always knew I can do better with my styling I didn't know how far I can take it. Check this post for a quick ramp-up on the topic.

https://towardsdatascience.com/style-pandas-dataframe-like-a-master-6b02bf6468b0

5 ways to follow publications in your field

Wed, 07 Aug 2019 12:12:41 GMT

This post was published on Medium

An important part of the data scientists and researchers’ life is to keep track of publications in their field. Depends on your field and needs publications range from papers in academic conferences and proceedings (some of them you can find as youtube videos), new technology and code packages, blog posts, etc. This post focus on who to keep track of academic research and innovation.

Follow the relevant conferences, journals — make sure you are familiar with the main conferences in your field (ee.g.List of Machine Learning and Deep Learning conferences in 2019 / 2020) and follow their publications. You can usually read the accepted papers in the conference website when the paper acceptance is published. Talk slides and videos are usually accessible a short while after the conference. Identifying the relevant conferences may require some initial effort but once you identified it, it is easy to get it going.
Google scholar e-mail alerts — track authors and \ or keywords you find relevant for you. E.g if you are interested in causal inference you would probably want to follow Judea Pearl. You can track new articles, citations and new articles related by author or keywords. I prefer to track only new articles because I found the benefit from citations and related articles low. You can also get email alerts by more complex queries. Set your alerts here.
arXiv E-Mail Alerting Service — arXiv provides a daily digest of new submissions by subject, it is less granular and less focused than google scholar but can give you access to the newest, hottest submissions. Subscribe to arXiv E-Mail Alerting Service here).
Follow blogs and publications of companies and research institutes which interest you — those are usually softer publications that give you a taste of the company’s recent advances and research. If this lights up your imagination, move on to reading the full paper. Examples of such blogs — facebook research blog, OpenAI blog, Google AI blog.
Social media — follow researchers which are relevant to your field in twitter, see the papers they publish and recommend, read the discussions they are involved in. Join facebook groups that discuss the topics you are interested in.

Now, you can lean back and enjoy the new ideas coming to you. The next challenge is to wisely invest your time and to pick the papers which will be most beneficial for you.

Junior Data Science — Choosing your first job

Fri, 26 Jul 2019 19:19:13 GMT

This post was publish on Medium

While there are many people who would like to become a data scientist and are looking for their first position, junior data science positions are rare. Data science positions range from very research oriented positions in companies which also publish in scientific conferences (quite rare) to positions which are more hands-on and involve lots of coding. (Junior) Data scientists also come from diverse backgrounds: recent grads (bsc, msc and PhDs in different fields), experienced developers which would like to learn new skills, retraining and so on.

While the junior data science positions are rare, it is important to make an accurate choice and avoid common pitfalls. This post was triggered by Ori Cohen’s post “Data-Science Recruitment — Why You May Be Doing It Wrong” which was oriented to the recruiting side. This post is for the data scientist who is looking for their first job. Here are few insights.

Don’t be the first data scientist in the company

This sounds like a very sexy position — you recently graduated from the university and you were able to impress a small startup with your skills. They offer you to be the first data scientist in the company, boom! You will be able to shape the methods, process and tools the right way, like you always envisioned!

״In theory, theory and practice are the same. In practice, they are not״(Benjamin Brewster).

Many practical tasks are not like in the textbook or in Andrew Ng’s course. You will most probably need guidance and advice from an experienced data scientist who already made her mistakes, is familiar with the data and with the product’s constraints and is simply more experienced. The skills you want to learn varies over time but it always a good idea always have someone around that you can learn from.

An additional issue is that small companies usually have little data, usually not enough to train models, and the data quality might also be an issue. This will require changes in the product which should be defined and implemented. As a junior data scientist it might be complicated to do both the technical part and the politics which is required for such a change.

How would you know you are interviewing for the first data science position:

You will be told so explicitly — “you will be our first data scientist”
None of your interviewers is a data scientist and the questions they ask don’t reflect a deep understanding of the topic.

People Don’t Quit Jobs — They Quit Bosses

And before quitting — people work for bosses.

Interviews are two-sided. The company interviews you, but you also interview the company. Does the product excites you? Do you think the company has the right values and culture fit for you? Would you like to work for this manager?

Most likely you will work closely with your manager and teammates. Did they impress you? Would you value their feedback?

In order to learn and improve, a lot of feedback and communication is required, especially when you are in a junior position. Are there regular 1:1s? Is there an on boarding plan? Do they participate in conferences \ is there an education budget? Does the company have the work-life-balance you are looking for?

During an interview, the interviewer might want to please you so if you’ll ask these questions directly they might answer what you expect to hear. Talking with teammates and other co-workers in the company can give you additional insights about the team and the company.

Tools and Technologies

If you mainly focus on research you might find this point secondary. However, for your next position, hands on experience may be required. Be sure to choose a place which uses reasonable technologies and not a niche, esoteric technologies. E.g using assembly for machine learning, working in mainframe environment, etc.

Current reasonable technology stack for data scientist includes : python (maybe scala, maybe R depends on your risk aversion) and scientific python packages (pandas, numpy, scipy, etc), cloud environment, some kind of database (postgres \ mysql \ elasticsearch \ mongodb).

Last but not least — choose something you are passionate about so you will be happy to go to work in the morning and dream about your code at night :)

Special thanks to Liad Pollak and Idit Cohen who made this text readable

5 interesting things (23/07/2019)

Tue, 23 Jul 2019 13:22:55 GMT

Five Talks from spaCy-IRL Worth Watching - great summarisation of 5 talks from spaCy-IRL conference which took place in Berlin in the beginning of July. The summarisations are very exact - not too deep, not too shallow and makes you want to watch the talks. From the meta perspective - a very nice connection between academia and industry leveraging ideas from academia to solve industry problems.

https://medium.com/@noa.kel/five-talks-from-spacy-irl-worth-watching-e6e505e0e272

King - Man + Woman = King ? In 2016 "Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings" was published and showed that the pre-trained word2vec model which was trained on Google News articles exhibited gender stereotypes to, “a disturbing extent.”. Apparently, according to "Fair is Better than Sensational:Man is to Doctor as Woman is to Doctor" at least some of the bias stems from optimisations \ restrictions done in order to present better results. Most significant one the answer to "a to b is like c to ..?" cannot be b. This does not mean that there is no bias, it only means that it was not measured and formalised correctly. This emphasises once again the need to understand the algorithms we use and their limitations.

https://blog.esciencecenter.nl/king-man-woman-king-9a7fd2935a85

Bonus - linear digression episode - Revisiting Biased Word Embeddings

10 tips for code review - code review can be a stressful task for both the reviewer and the person her work is being reviewed. This post is from the reviewer point of view, how to make this process more efficient and constructive to both sides. A good follow up post would be how to listen and reach to code review. From my experience, many times it is a boiling point for relationships inside teams and can break teams when not done correctly.

https://developers.redhat.com/blog/2019/07/08/10-tips-for-reviewing-code-you-dont-like/

How to label data - if you ever did a data science project you know that obtaining tagged data is a real hassle. You often discover that you don't have enough data, the tagging is not what you need, etc. This guide will help you avoid pitfalls when issuing a labelling project.

https://www.lighttag.io/how-to-label-data/

Data-Science Recruitment — Why You May Be Doing It Wrong - post by data science team lead in Zencity regarding do's and don't do in the interviewing process for data scientists. In the last few years I widnessed many of this flaws - asking non relevant riddles, given a very long home exercise, not well defined with doubtful data. I would like to emphasise for candidates, specially junior candidates, that if you have doubts during the interview process consider looking for another place.

https://towardsdatascience.com/data-science-recruitment-why-you-may-be-doing-it-wrong-b8e9c7b6dae5

5 interesting things (26/06/2019)

Wed, 26 Jun 2019 13:22:54 GMT

Checklist for debugging neural networks - well written trouble shooting for neural networks models which is not language or framework specific! https://towardsdatascience.com/checklist-for-debugging-neural-networks-d8b2a9434f21

Why Software Projects Take Longer Than You Think A Statistical Model - great post about a problem we all face. Usually we try to solve it using "instrumental changes" - changing methods \ processes \ ... . This post tries to show that there is more to it than just the behavioural change.

https://erikbern.com/2019/04/15/why-software-projects-take-longer-than-you-think-a-statistical-model.html

(and linear digressions episode about it - http://lineardigressions.com/episodes/2019/4/21/using-statistics-to-understand-why-your-software-projects-are-always-running-late)

Google What-If-Tool (WIT) - A nice tool by Google that was released few month ago. The terminology is actually a bit misleading and counterfactuals don't carry the meaning they have in causal inference. It is more like matching with two possible distance matrices - L1 and L2.

https://ai.googleblog.com/2018/09/the-what-if-tool-code-free-probing-of.html

causallib - New python causal inference package from IBM

https://github.com/IBM/causallib

There is also a python causal inference package from Microsoft which was released about a year ago - https://github.com/Microsoft/dowhy.

A Visual Intro to NumPy and Data Representation - What can I say, I really like Jay's guides -

https://jalammar.github.io/visual-numpy/

4 insights from BDHW19

Tue, 28 May 2019 10:54:09 GMT

This week I attended BDHW19 - Big data in Health Care which was hosted by the Weizmann Institute of Science in collaboration with Nature Medicine. The conference had a great line up of speakers - leading researchers in the field from academia, industry and HMO's.

There were few ideas and themes that were mentioned several times from different angles and I would highlight few of them.

(all sessions were recorded and I'll add a link once they are online)

EHR data in Israel - By law, every Israeli resident must be registered with one of the HMO's. The HMO's in Israel are in a special position were the where they are run as non-for-profit organizations and are prohibited by law from denying any Israeli resident membership. Israelies HMO's hold EHR data from the mid nineties which means that the biggest HMO (Clalit) have longitudinal data of over 20 years for 4.5m heterogenic patients. Together with greater researchers and collaboration with the academia this enables amazing research which hopefully later propagates and influence our daily life.

See more in - Clalit publications, Maccabi publications

The Israeli AI Healthcare Startup Landscape of 2018 - https://www.startuphub.ai/israeli-ai-healthcare-startups-2018

Deployment of HC models - while there are great result and tools developed on research the way to deploy those models, use the new ideas is long and contains many obstacles. Only very few models really turned into health care products - alert system, treatment guide lines, bio markers, personalized medicine, etc. Few caveats in the way are interpretability, robust machine learning and causality. We must keep in mind that eventually our research should affect the end users - clinicians, patients, etc.

5 > 11

Thu, 17 Jan 2019 09:22:48 GMT

But it takes 11 minutes to read the tutorial...

5 interesting things (17/01/2019)

Thu, 17 Jan 2019 08:39:13 GMT

How to Grow Neat Software Architecture out of Jupyter Notebooks - jupyter notebooks is a very common tool used by data scientist. However, the gap between this code to production or to reusing it is sometimes big. How can we over come this gap? See some ideas in this post. https://github.com/guillaume-chevalier/How-to-Grow-Neat-Software-Architecture-out-of-Jupyter-Notebooks High-performance medicine: the convergence of human and artificial intelligence - a very extensive survey of machine learning use cases in healthcare. https://www.nature.com/articles/s41591-018-0300-7 New Method for Compressing Neural Networks Better Preserves Accuracy - a paper by Amazon Alexa team (mainly). Deep learning models can be huge and the incentive of compressing them is clear, this paper show how to compress the networks while not reducing the accuracy too much (1% vs 3.5% of previous works). This is mainly achieved by compressing the embedding matrix using SVD. https://developer.amazon.com/blogs/alexa/post/a7bb4a16-c86b-4019-b3f9-b0d663b87d30/new-method-for-compressing-neural-networks-better-preserves-accuracy Translating Between Statistics and Machine Learning - different paradigms sometimes use different terminology for the same ideas. This guide tries to bridge the terminology gap between statistics and machine learning. https://insights.sei.cmu.edu/sei_blog/2018/11/translating-between-statistics-and-machine-learning.html Postmake - "A directory of the best tools and resources for your projects". I'm not sure how best is defined but samplig few categories it seems good (e.g. development categorty is pretty messy including github, elasticsearch and sublime together). I liked the website design and the trajectory. I do miss some category of task managment ( couldn't find Jira and any.do is not really a calender). It is at least good resource for inspiration. https://postmake.io

5 interesting things (4/11/2018)

Sun, 04 Nov 2018 18:45:28 GMT

Deep density networks and uncertainty in recommender systems - Yoel Zeldes and Inbar Naor from Taboola engineering team published a series of posts (4 so far) about uncertainty in models - where this uncertainty comes from, how one can explore and use this uncertainty, etc. This post series relates to a paper they present in the workshop in this year KDD conference.

First post - https://engineering.taboola.com/using-uncertainty-interpret-model/

Paper - https://arxiv.org/abs/1711.02487

Decision tree visualization - this post will be part of The Mechanics of Machine Learning by Terence Parr and Jeremy Howard. The post discusses the creating of dtreeviz from several aspects - considerations regarding visualizing decisions trees, comparison to current tools, implementation details, etc. Fascinating read.

http://explained.ai/decision-tree-viz/index.html

The Tale of 1001 Black Boxes - many words were already spilled about the model Amazon used trying to automate their HR system. I like this one as I believe it explains the pitfalls clearly even to someone how is not an ML professional and it tries to grow from this point.

https://towardsdatascience.com/the-tale-of-1001-black-boxes-62d12b5886aa

Bonus point - https://twitter.com/kldivergence/status/1050119653849092096?s=21

Lessons Learned from Applying Deep Learning for NLP Without Big Data - in the last 2 years everyone are doing deep learning but to be honest one of the very common issues in the industry is not having enough labeled data and thus deep learning can not always being applied. This post suggest few techniques to overcome not having enough data for NLP tasks.

https://towardsdatascience.com/lessons-learned-from-applying-deep-learning-for-nlp-without-big-data-d470db4f27bf

Machine Learning for Health Care course - a paper a day keeps the doctor away. Not this doctor ;-)

Syllabus of Princeton Machine Learning for Health Care course (COS597C) given by Barbara Engelhardt. The reading list is very varied (from NLP to vision through reinforcement learning) and interesting. I definitely add at least some of those papers to my queue.

http://www.cs.princeton.edu/~bee/courses/cos597c.html

5 interesting things (13/08/2018)

Mon, 13 Aug 2018 08:49:59 GMT

JQ cook book - I find myself using JQ quite often and sometimes to more complex things than just filtering fields.

https://github.com/stedolan/jq/wiki/Cookbook

Bonus point - list of text-based file formats and command line tools for manipulating each - https://github.com/dbohdan/structured-text-tools

Missingno - Missing data visualization module for Python. This package offers a variety of visualization to understand the missing data in your data set and the correlations between the absent of different properties.

https://github.com/ResidentMario/missingno

Add time order in Recommendation system - the meaning of time order in this context is item x should be followed by item y. E.g for tv series - chapter 1 should be viewed before chapter 2. I don’t know if it is state of the art work in this domain but it is nice and should be relatively easy to implement when the prerequisite graph is unknown.

https://medium.com/@jujukala/how-to-use-time-order-in-machine-learning-recommendations-for-education-97091d2ab138

Cognitive bias are everywhere - and this time who they affect you management performance. Referring to the last 2 points in the summary - “Establish trust and openness with your peers and reports” - I’m a big believer in 1:1s, I found this resource bundle here. “Understand motivational theory, especially intrinsic motivation” - maybe the most important thing I learned being a scout leader is that every person have different motivation, you cannot lead others by what motivate you. Understanding this made a big change on how I view the world.

https://medium.freecodecamp.org/cognitive-bias-and-why-performance-management-is-so-hard-8852a1b874cd

Want to Improve Your Productivity at Work? Take a Cooking Class - on general I really like when interdisciplinary ideas mix and this is an interesting thought about the topic. The point that was most interesting for me was “Set your mise-en-place”. I see it a bit different \ wider from the writer - as a manager you should sometimes prepare the “mise-en-place” for your team. If they need to integrate with external service - take care of the NDA, API documentation, etc. Requirements and design can also sometimes viewed as “mise-en-place” for developers.

https://medium.com/forbes/want-to-improve-your-productivity-at-work-take-a-cooking-class-37ac08bf2f26

Bayesian machine learning in Python: A\B testing

Mon, 06 Aug 2018 14:14:49 GMT

I have recently took "Bayesian machine learning in Python: A\B testing" course in Udemy. My notes from the course can be found here. It is mainly written for myself for easier future access and some links I browsed while taking the course, I hope it will be beneficial to others as well.

Geekness killed the cat (and others)

Mon, 04 Dec 2017 19:03:37 GMT

https://mathwithbaddrawings.com/2017/11/29/epitaphs-in-the-graveyard-of-mathematics/ And a follow up - https://gorelik.net/2017/12/02/epitaphs-in-the-graveyard-of-mathematics/

Fun Facts: Automatic Trivia Fact Extraction from Wikipedia

Sat, 21 Oct 2017 11:49:10 GMT

Authors: David Tsurel, Dan Pelleg, Ido Guy, Dafna Shahaf Article can be found here Trivia facts can drive users engagement, But what are trivia fact? Is the fact “Barack Obama is part of the Obama family” a trivia fact?
Is the fact “Barack Obama is Grammy Award winner” a trivia fact? This paper tackle the problem of automatically extracting trivia facts from Wikipedia. In this paper Tsurel et al. focussed on exploiting Wikipedia categories structure (i.e. X is a Y). Categories represent set of articles with common theme such as “Epic films based on actual events”, “Capitals in Europe”, “Empirical laws”. An article can have several categories. The main motivation to use categories and not free text is that categories are cleaner than sentences and capture the essence of the sentence better. According to Miriam-Webster dictionary a trivia is:

unimportant facts or details
facts about people, events, etc., that are not well-known

The first path Tsurel et al. tested was to look for a small categories a”presumably, a small category indicates a rare and unique property if an entity, and would be an interesting trivia fact”. This path proved to be too specific e.g “Muhammad Ali is an alumni of Central High School in Louisville, Kentucky”. [TR] As commented in the paper this fact is is not a good trivia fact because the specific high school has no importance to the reader and or to Ali’s character. But, as stated later - when coming to personalizing trivia facts there maybe readers which find this fact interesting (e.g other alumni’s of this high school). This led Tsurel et al. to the first required property of trivia fact - surprise. Surprise Surprise reflects how unusual the article with respect to the category. So they would like to define similarity matrix between article a and category C. A category is a set of articles therefore the similarity is defined as:

$latex similarity(a, C) = \sigma(a, C)=\frac{1}{|C|-1}\sum_{a \neq a' \in C}\sigma(a, a')$

Surprise is defined as the inverse of the average similarity -

$latex surp(a, C)=\frac{1}{\sigma(a, C)}$

Example of results for this measure for Hedy Lamarr - As you can see in the example above the surprise factor itself is not enough as it does not capture other aspects in Hedy Lamarr’s life (e.g. she invented radio encryption!). Examining those categories and seeing that they are very spread led the team to the define the cohesiveness of category. Cohesiveness Cohesiveness of a category measures the similarity between items in the same category. Intuitively if an item is not similar to the other items in the category it might indicate that it is a trivia fact (or as mention later in the paper - detect anomalies). Practically speaking the cohesiveness if category C is defined as the average similarity between each pair of articles in the category.

$latex cohesive(C)=\frac{1}{{|C| \choose 2}}\sum_{a \neq a'} \sigma(a, a')$

Hedy Lamarr's results w.r.t to cohesiveness - Tying it together The trivia score of article a to category C is define as:

$latex trivia(a, C)=cohesive(C) \cdot surp(a, C) = \frac{cohesive(C)}{\sigma(a, c)}$

Interpret trivia score:

Around one - this means that $latex cohesive(C) \approx \sigma(a, C)$. Meaning - the article is typical for the category, i.e. similar to other articles in the category.
Much lower than one - “the article is more similar to other articles in the category than the average”. That means the article is a very good representative of the category.
Higher than one - the article is not similar to the category, i.e is an “outsider” which make it a good trivia candidate.

Article similarity Standard similarities methods don't fit this case from 2 mains reasons -

The authors look for broad similarity and not details similarities.
Term frequency vector capture semantic similarity which sometimes get lost even after using normalization techniques.

Algorithm

Describe each article by the top K TF-IDF in the text. The TF-IDF is trained on a sample of 10,000 wikipedia articles after stemming, stop-words removal and case folding. K=10 in their settings. The table below show the results for the articles "Sherlock Holmes", "Dr. Watson" and "Hercule Poirot". As one can see it captures the spirit of the things but there are not exact matches.

To answer the exact match problem the authors used Word2Vec pre-trained model from Google News.
- $latex T_1$ and $latex T_2$ are the set of the top K TF-IDF term for articles $latex a_1, a_2$ respectively.
- For each term in $latex T_1$ find the most similar term in $latex T_2$ based on Word2Vec pre-trained model (and vice versa) and sum those similarities.

$latex \sigma(a_1, a_2)=\frac{1}{Z}\sum_{i=1}^K w_{i}\cdot(max_{1 \leq j \leq K}\sigma(T_1[i], T_2[j]) +max_{1 \leq j \leq K}\sigma(T_2[i], T_1[j]))$

where $latex w_i=K-i+1$ and $latex Z=2 \cdot { K+1 \choose 2}$ Further optimization on the computation such as caching, comparing only to subset of articles and parallel computation can be done when coming to implement this algorithm in production settings. Evaluation The authors evaluated their algorithm empirically against -

Wikipedia Trivia Miner - "A ranking algorithm over wikipedia sentences which learns the notion of interstingness using domain-independent linguistic and entity based features."
Top Trivia - highest ranking category according to the paper algorithm.
Middle-ranked Trivia - middle-of-the-pach ranked categories according to the paper algorithm.
Bottom Trivia - lowest ranked categories according to the paper algorithm.

The authors crawled wikipedia and created a dataset of trivia facts for 109 articles. For each article they created a trivia fact for each algorithm. The textual format was "a is in the group C". Trivia Evaluation Study Using the trivia facts above each fact was presented to 5 crown workers yielding 2180 evaluations. The respondents were asked to agree \ disagree the facts according to the following statements or note that they don't understand the fact:

Trivia worthiness - "This is a good trivia fact".
Surprise - "This fact is surprising"
Personal knowledge - "I knew this fact before reading it here"

The score of a fact was determined by the majority vote. Result -

The top trivia facts are significantly better than the WTM facts
The consensus on the trivia worthiness of the top facts compared to the WTM facts is higher (32.8% vs 11.9%).

[caption id="attachment_1155" align="aligncenter" width="344"] Is it trivia worthy fact?[/caption] [caption id="attachment_1163" align="aligncenter" width="329"] Is it a surprising fact?[/caption] [caption id="attachment_1166" align="aligncenter" width="300"] Did you know this fact before?[/caption] Engagement Study In this part the team used google ad to tie trivia facts to searches and analyzed the bounce rate and time on page for the collected clicks (almost 500). Results -

CTR was not significantly different than reported results in the market (0.8) - i.e does not indicate willingness of users to explore trivia facts.
Bounce rate (time of page < 5 seconds) for bottom trivia was 52%, for WTM facts 47% and for top trivia 37%.
Average time on page was significantly better in top trivia comparing to bottom trivia (48.5 seconds vs 30.7) but was not significant comparing to WTM (43.1 seconds).
One reason for people to spend time on WTM fact pages was because the presented sentences ware longer than the sentences presented for the top trivia and had a higher change of being confusing so people take time to understand them.

Discussion and Further work Limitation - the algorithm works well for human entities but worse on other domain such as movies and cities. Future work -

Better phrasing of the trivia facts - instead of “X is a member of group Y” —> “Obama won Best Spoken word Album Grammy Awards for abridged audiobook versions of Dreams from My Father in February 2006 and for the Audacity of Hope in February 2008”.
Turning trivia facts to trivia questions - for the example above generate a question of the form - “Which US presider is a Grammy award winner?” And not “Who won a Grammy award” or “What did Barack Obama win?”

[TR] - this would require additional notation of a good trivia question. The “good” question in this example is interesting since it involves a contrast between two categories.

Other applications -

Anomaly detection - surprising facts are sometimes surprising because they are wrong. Using this algorithm we can clean those and improve Wikipedia reliability.
Predict most surprising article in a given category
Improve search experience by enriching result with trivia facts

[TR] - Improve learning experience on learning platforms by enriching the UI with trivia facts.

Extensions -

Personalized trivia score - as commented above, different reader can find different facts more \ less interesting (see here) so the score should be personalized and take into account different properties of the reader such as demographic and even more temporal like mood.
[TR] - Additional extensions involve trivia facts between entities such as "Michelle Obama and Melania Trump are in the same height", "X and Y were born in the same date".

Map Spark UDAF (Java)

Mon, 10 Jul 2017 17:06:46 GMT

I run Spark code on Java. I had data with the following schema - [code language="bash"] root |-- userId: string (nullable = true</span> |-- dt: string (nullable = true)</span> |-- result: map (nullable = true)</span> | |-- key: string | |-- value: long (valueContainsNull = true) [/code] And I wanted to get a single record for a user which has the following schema - [code language="bash"] root |-- userId: string (nullable = true)</span> |-- result: map (nullable = true) | |-- key: string | |-- value: map (valueContainsNull = true) | | |-- key: string | | |-- value: long (valueContainsNull = true) [/code] Attached the user defined aggregation function I wrote to achieve it. Before that - [code language="bash"] MergeMapUDAF mergeMapUDAF = new MergeMapUDAF(); df.groupBy("userId").agg(mergeMapUDAF.apply(df.col("dt"), df.col("result")).as("result")); [/code] https://gist.github.com/tomron/36fd3c1b41169fc40acaeb4dbe95067d

Code Challenges Anti-Patterns

Wed, 01 Mar 2017 18:03:14 GMT

Code challenges are a common tool to evaluate candidate ability to develop software. Of course there are other indicators such as - blog posting, open source involvement, github repository, personal recommendations, etc. Yet, code challenges are frequently used. I recently got to check some code challenges and was surprised from some of the things I found there. Here are my anti-patterns to code challenges -

Call a file \ process on your local machine

[sourcecode language="python" wraplines="false" collapse="false"] for line in open('/Users/user/code/data.csv'): print ('No, No, No!') [/sourcecode] The person how check your code challenge cannot just run the code since she will get a file not found error or similar and will have to find where you call the file and why. If you need to call some resource (file, database, etc.) you can pass it as a command line argument, put it in a config file, use environment variable.

Reinvent the wheel

Write everything by yourself. Why you crowd wisdom or mature project which are already debugged and tested when you can write everything by yourself the way you like it with your own unique bugs? Unless you were told otherwise many times there is already a package \ library \ API \ design pattern which is doing part of what you need. E.g if you need to fetch data from Twitter there is Twitter API and there Twitter clients in different languages. You definitely don't need to crawl twitter and process the HTML.

Don't write a README file

No need to write a README file. Whoever is reading your code is a professional in the tech stack you chose and will immediately know how to start your project, which dependencies are there, which environment variables are needed, etc. The goal of README in this context is to explain how to run the code, what is inside the package and further considerations \ assumptions \ choices you did while working on this challenge. A detailed README is always priceless and specially in this context when you don't always have a direct communication with the candidate. System diagram \ architecture chart is also recommended when relevant. If you have further notes such as ideas on how to expand this system, what would you do next, etc. I would put it in IDEAS files (e.g. IDEAS.md) and also link the README from it.

Don't write tests

This is actually the part which highlight your genius. You don't need to test your code because it is perfect. Seriously, testing plays a big role in software development and making sure the code you wrote work as expected. As a viewer from the side it also give me a clue how pedant you are and how much you care about the quality of your work. This is my first impression of your work, don't make it your last.

ZIP you code

This is how we deploy and manage versions in our company - I just send my boss a zip file, preferably via slack. I expect to get a link to a git repository (if you want to be cautious you can use private repos e.g. by bitbucket). In this link I can see the progress you made while working on the challenge, your commit messages (also a signal on how pedant you are). As a candidate you also get to demonstrate your skills in version control system in addition to your coding skills.

Wrap Up

There are of course many other things I can point to such as general software development practices - magic numbers, meaningful names, spaghetti code, etc. But as said - those are general software development skills that one should use everyday, the anti-patterns stated above are IMO specially important in the case of code challenges.

SO end of year surveys

Thu, 16 Feb 2017 15:17:15 GMT

Recently Stack Overflow published few posts comparing the usage of Stack Overflow between different segments \ scenarios:

How Do Students Use Stack Overflow?
What Programming Languages Are Used Most on Weekends?
Women in the 2016 Stack Overflow Survey

Few comments regarding those posts - How Do Students Use Stack Overflow?

"R and MATLAB are pretty consistent throughout the year" - those are research related programming languages and it make sense that student to advanced degrees use them all around the year. Python is also used a lot in research and one can see that the decrease during the the summer vacation is smaller than the decrease in Java and C.

Public universities vs Private universities - the list of the most common tags in the public universities include beside Microsoft products also - while-loop, methods, input, linked-list, if-statement, loops, etc. Those are all theoretical \ foundation related concepts. On the other hand the private list contain not theoretical concepts beside web-scraping, but stack specific products, e.g - mongodb, iphone, aws, flask, heroku, etc. For me that flags either lecturers on private universities explain better theoretical concepts and therefore the students don't ask questions about them or that private universities invest more on preparing the students to the labor market and therefore use more industry products.

This post end with a chart of the Universities that asked the most R questions relative to other technologies - "The most R-heavy schools ... and some are recognizable as having prominent statistics programs". This brings me to another analysis I would like to see - what are the majors of the students who asked those questions?

And of course segmenting it together with the other analysis - e.g. how female students uses stackoverflow versus of male students use it. In which universities students work hardest in the weekends?

What Programming Languages Are Used Most on Weekends?

Which tag's weekend activity has decreased the most? As SO suggest the usage of some of the technologies decreased on weekends because the usage of those technologies decreased on general such as SVN and probably RoR3 and visual-studio-2012. On the other hands, other technologies such as Scala and maybe Azure become more popular and mainstream so it is no longer only geeks technologies used in weekends. To confirm this claim it would be interesting to see if the currently increased weekend technologies (e.g. android-fragments, android-layout, unity3d, etc.) will become mainstream in few years.

It is also interesting to see if the weekend technologies are also technologies used at after work hours (this is of course timezone dependent).

Also interesting to see the difficulty levels of weekend posts versus workdays posts. Does users use weekend to learn new technologies and therefore ask relative basic questions or they dive in to a technology and ask advanced questions. The difficulty of the questions can be evaluated by the time it took the get answer (more complex question -> longer time to get answer), up-votes, etc.

Women in the 2016 Stack Overflow Survey

Those surveys are biased to begin with. Having said that, the gap between people who identify ask female and people who identify as male who answered cries out loud. It is even bigger than the gap in the labor market. Where does this gap comes from? Maybe women are just better in search as many of the SO questions are duplicated ;-)

This is noted also in the post itself. It is also interesting to see which other populations were under represented in the survey.

An interesting analysis I would like to see is the sentiment (aggressiveness vs calm \ informative in the answers) segmented by the repliers gender. Maybe also as a function of the asker gender. I must say the personally never noticed the asker's gender.

Davies-Bouldin Index

Wed, 30 Nov 2016 19:06:54 GMT

TL;DR - Yet another clustering evaluation metric Davies-Bouldin index was suggested by David L. Davies and Donald W. Bouldin in "A Cluster Separation Measure" (IEEE Transactions on Pattern Analysis and Machine Intelligence. PAMI-1 (2): 224–227. doi:10.1109/TPAMI.1979.4766909, full pdf) Just like Silhouette score, Calinski-Harabasz index and Dunn index, Davies-Bouldin index provide an internal evaluation schema. I.e. the score is based on the cluster itself and not on external knowledge such as labels. The Silhouette score reflects how similar a point is to the cluster it is associated with. I.e .for each point with compute the average distance of the point from the points in the nearest cluster minus the average distance of the point from the points in its own cluster divided by the maximum between those distances. The overall score is the average of the score per point. The Silhouette score is bounded from -1 to 1 and higher score means more distinct clusters. The Calinski-Harabasz index compares the variance between-clusters to the variance within each cluster. This measure is much simpler to calculate then the Silhouette score however it is not bounded. The higher the score the better the separation is. The intuition behind Davies-Bouldin index is the ratio between the within cluster distances and the between cluster distances and computing the average overall the clusters. It is therefore relatively simple to compute, bounded - 0 to 1, lower score is better. However, since it measures the distance between clusters' centroids it is restricted to using Euclidean distance function. Silhouette score and Calinski-Harabasz index were previously implemented and are part of scikit-learn and I implemented Davies-Bouldin index. Hopefully this will be my first contribution to scikit-learn, my implementation is here.

5 Berlin Data Native 2016 Highlights

Mon, 31 Oct 2016 11:27:07 GMT

Super Mario from Microsoft (Daniel Molnar) - Data Janitor 101, one of the best reasoned talks I heard for a long time.
Andrew Clegg, data scientist @ Etsy gave an historic review on Semantic Similarity and Taxonomic Distance and how it is used in Etsy. Slides are here.
Topic Modeling on Github repositories - presented by Vadim Markovtsev from source{d}. Why is that interesting? Few possible evolutions - code quality, natural language to code. Also interesting direction to have a look on - bigartm.
Data Thinker - Is it a new role? Is it just a buzzword? Klaas Bollhoefer from the Unbelievable Machine presented their ideas.
Databases - The Choice is Yours - Philipp Krenn from Elastic presented his view on the databases field.

Data Natives Berlin 2016 (1st day)

Thu, 27 Oct 2016 15:41:26 GMT

[gallery ids="889,890" type="rectangular"]

From Philipp Krenn's, Developer Advocate at Elastic, "Databases - The Choice is Yours" talk.

Detecting Data Errors: Where are we and what needs to be done?

Sun, 02 Oct 2016 14:10:24 GMT

My summary and notes for "Detecting Data Errors: Where are we and what needs to be done?" by Ziawasch Abedjan, Xu Chu, Dong Deng, Raul Castro Fernandez, Ihab F. Ilyas, Mourad Ouzzani, Paolo Papotti, Michael Stonebraker, Nan Tang Proceedings of the VLDB Endowment 9.12 (2016): 993-1004. Paper can be found - here In this paper the group of researchers evaluate several data cleaning tools to detect different types of data errors and suggest a strategy to holistically run multiple tools to optimize the detection efforts. This study focus on automatically detecting the errors and not repair them since automatically repairing is rarely allowed. Current status - current data cleaning solutions are usually belong to one or more of the following categories:

Rules based detection algorithms - the user specify set of rules such as: not null, functional dependencies, user defined function that the data must obey and the data cleaner find any violation. Example: NADEEF.
Pattern enforcement and transformation tools - tools in this category discover either syntactic or semantic patterns in the data and detect those errors. Example:OpenRefine, Data Wrangler, DataXFormer, Trifacta, Katara.
Quantitative error detection algorithms - find outliers and glitches in the data.
Record linkage and de-depulication algorithms - identify data which refer to the same entity and is not consistent \ appear multiple times. Examples: Data Tamer, TAMR.

Evaluation of tools

Precision and recall of each tool
Errors detected when applying all the tools together
How many false positives are detected as we would like to minimize the human effort.

Error types

Outliers include data values that deviate from the distribution of values in a column of a table.

Duplicates are distinct records that refer to the same real-world entity. If attribute values do not match, this could signify an error.

Rule violations refer to values that violate any kind of integrity constraints, such as Not Null constraints and Uniqueness constraints.

Pattern violations refer to values that violate syntactic and semantic constraints, such as alignment, for matting, misspelling and semantic data types.

(p. 995) Some errors overlap and fit into more than one category. [TR] - are those all the error types which exist? What about correlated errors between several records? Data sets [TR] - there are many data sets specific details in the paper. As I am more interested in the ideas those details will be omitted here. [TR] - the data evaluated relatively small datasets with small number of columns. It would also be interesting to evaluate it bigger and more complex datasets, e.g. wikidata. [TR]- consider the temporal dimension of the data. I.e some properties may have expiration date which other not (e.g birth place never changes while current location changes). Data cleaning tools

	DBoost	DC-Clean	OpenRefine	Traficata	Pentaho	Knime	Katara	TAMR
Pattern violation			+	+	+	+	+
Constraint violations		+
Outliers	+
Duplicates								+

DBoost - use 3 common method for outlier detection - histograms, Gaussian and multivariate Gaussian mixtures. The UVP of this tool is decomposing types into their building blocks. For example expanding dates into day, month and year. DBoost require configuration such as number of bins and their width for histograms and mean and standard deviation for Gaussian and GMM.
DC-Clean - focus on denial constraints and subsume the majority of the commonly used constraint languages. The collect if denial constraints was designed for each data set.
OpenRefine - can digest data in multiple formats. Data exploration is performed through faceting and filtering operations ([TR] - reminds DBoost histograms)
Trifacta - commercial product which was developed from DataWrangler. Can predict and apply syntactic data transformation for data preparation and data cleaning. Transformations can also involve business logic.
Katara - uses external knowledge bases, e.g. Yago in order to detect errors that violate a semantic pattern. It does it by first identifying the type of the column and the relations between two columns in the data set using a knowledge base.

[TR] - assumes that the knowledge base is ground truth. We need to doubt this as well.

Pentaho - provide graphical interface for data wrangling and can orchestrate ETL processes.
KNIME - focuses on workflow authoring and encapsulating data processing tasks with some machine learning capabilities.
TAMR - uses machine learning models to learn duplicate features through expert sourcing and similarity metrics.

Combination of Multiple tools

Union all and Min-K -
- Union all - takes the union of the errors emitted by all tools (i.e k=1)
- Min-k - error detected by at least k tools.
Ordering based on Precision
- Cost model -
  - C - cost of having a human check a detected error
  - V - Value of identifying a real error (V > C otherwise make not sense).
  - P - Number of true positives
  - N - Number of false positives
  Total value should hold - P * V > (P +N) * C => P/(P+N> > C/V. P/(P+N) is the precision. Therefore if the model precision is less than C/V we should not run it. Model precision can be evaluated by sampling the detected errors. "We observed that some tools are not worth evaluating even if their precision is higher than the threshold, since the errors they detect may be covered by other tools with higher estimate precision (which would have been run earlier)." (p. 998) [TR] - not always the cost and value can be estimated correctly and easily and not all the errors have the same cost and value.
- Maximum entropy-based order selection - the algorithm estimate the overlap between the tool results and picks the tool with the highest precision to reduce the entropy. Algorithm steps:
  1. Run individual tool - run each tool and get the detected errors.
  2. Estimate precision for each tool by checking samples
  3. Pick a tool which maximize the entropy among the unused tools so far - picks the one with the highest estimated precision on the sample and verifies its detected errors on the complete data that have not been verified before.
  4. Update - update the errors that were detected by the chosen tool in the last step and repeat last two steps.

Discussion and Improvements

Using Domain specific tools - for example AddressCleaner. [TR] - only relevant when such a tool exists and is easy \ cheap to use.
Enrichment - rule-based systems and duplicate detection strategies can benefit from additional data. Future work would consider data enrichment system.

Conclusions

No clear winner - different tools worked well on different data set mainly due to different error types distribution. Therefore a holistic strategy must be used.
Since there are errors which overlaps one can order the tools to minimize false positives. However the ordering strategy is data set specific.
Yet, not 100% of the errors are detected. Suggested way to improve it is -
1. Type-specific cleaning - for example date cleaning tools, address cleaning tools etc. Even those tools are limited in their recall.
2. Enrichment of the data from external sources

Future work

A holistic combination of tools - algorithms for combining tools. [TR] - reminds ensemble methods from Machine learning.
Data enrichment system - adding relevant knowledge and context to the data set.
Interactive dashboard
Reasoning on real-world data

5 interesting things (28/08/2016)

Sun, 28 Aug 2016 10:57:15 GMT

Joel test for Data Science - the inspiration and adjustment to data science done in Domino, both were interesting reads for me.

https://blog.dominodatalab.com/joel-test-data-science/

What I Wish I Knew About Data For Startups - I speciialy related to documenting and testing events tracking. I was surpsirsed about how little this topic and its best practices are discussed.

http://www.jeannicholashould.com/what-I-wish-I-knew-about-data-for-startups.html

A Paper a week keeps the doctor a way? Git repository by Shagun Sodhani who summarizes and sometimes comment those papers.

https://github.com/shagunsodhani/papers-I-read

Inside Elasticsearch - 3 parts blog series about how elasicsearch work.

https://blog.insightdatascience.com/anatomy-of-an-elasticsearch-cluster-part-i-7ac9a13b05db#.vdqzkhrle

Is it brunch time? Someone is finally asking the important questions in life... defining brunch time using on twitter tweets timing. I liked the method used to define the interval of lunch which requires no more than high school math and also the question. From anthropological point of view it is interesting for me to see if different locations in the world define breakfast \ brunch \ lunch in different time. I.e in Spain people each lunch at 14 while in Israel they eat at 12. Also maybe interesting to compare tweets with pictures to tweets without pictures. My assumptions is that tweet without pictures are people on the way to brunch and would be a bit earlier comparing to tweets with pictures which are tweeted during the brunch.

https://medium.com/@bjacobso/is-it-brunch-time-ffe3adf485d8#.ypfoami7j

Bonus track - Prediction.io is now an Apache project (incubating). After few changes in the last few month - being bought by Sales Force, fork by ActionML, etc. Sales Force donated Predicion.io to ASF. Looking forward for the future of this project.

http://predictionio.incubator.apache.org/

5 Python NLP pacakges

Sun, 19 Jun 2016 20:42:34 GMT

NLP is a broad term which contains many types of question and challenges such as - language detection, Part-of-Speech tagging, relation extraction, named entity recognition, OCR, speech recognition, sentiment extraction and many more.

There are of course, several Python libraries which try to tackle some of those problems. This post aims to provide a short overview of those packages.

NLTK

Probably the oldest and the most known package in this area. Started in 2001 in Penn State Computer Science department the Natural Language Toolkit aims to support scientific research. Last stable release was released at the beginning of April, so the project is live and develops all the time.

The NLTK package includes wide variety of modules including - text tokenization, pos tagging, text classification, sentiment analysis, etc.

The downside of this package is that it is many times ungainly, heavy, complicated. It is more of academic level than on industry level. If you have lot's of text to analyze, specially if it is complicated, expect it to run for ages.

http://www.nltk.org/

http://www.nltk.org/book/

Spacy.io

Possibly NLTK strongest competitor with the goal of creating production level code.

Main capabilities include tokenizing, tagging, parsing, entity recognition and pattern matching. For now only supports English and German.

It is faster and suits more to industrial needs, but the community is still small comparing to NLTK and the features are also behind.

https://spacy.io/

gensim

'Topic modelling for humans"

Implements top notch algorithms focusing on topic modeling, documents ranking and significant terms identification, e.g. - tf-idf, word2vec, doc2vec, latent Dirichlet allocation (LDA), latent semantic analysis (LSA). Some of the algorithms can be run in a distributed manner.

Uses NumPy and SciPy for efficient processing.

https://radimrehurek.com/gensim/

LDIG

While NLTK old the philosophy of "one package rule them all". LDIG does one thing - language detection for short text, i.e n-grams distribution are based on twitter and meant to analyze texts with at least 3 words. Rather then relatively short texts.

They 99.1% accuracy over 17 languages. From my experience the accuracy with ldig was a bit lower (around 80%). However, still relatively good specially for Latin languages.

https://github.com/shuyo/ldig

https://shuyo.wordpress.com/2012/02/21/language-detection-for-twitter-with-99-1-accuracy/

scikit-learn

Scikit-learn is one of the biggest machine learning pacakges in Python. As NLP is one application of machine learning it features also specific modules to deal with text. So once you know the mathematical background of the algorithm you want to use you can use scikit-learn implementation.

Text feature extraction - http://scikit-learn.org/stable/modules/feature_extraction.html#text-feature-extraction

LDA - http://scikit-learn.org/dev/modules/generated/sklearn.decomposition.LatentDirichletAllocation.html

PyData Berlin 2016 #pydatabln

Sat, 21 May 2016 18:15:48 GMT

I got a diversity scholarship from Num Focus to attend the PyData Berlin event. Num Focus is an NGO which supports open source data science projects among them - Jupyter, matplotlib, Numpy, pandas etc.

This post is not a summary of the events or of the talks that I attended in but rather hints to a subset of the talks.

Keynote - Olivier Grisel (Inria)

Grisel talked about the Evolution of predictive modeling and scaling predictive modeling. Why do we need to scale predictive modeling -

I\O intensive operations - e.g feature engineering and model serving.
CPU intensive operations - e.g hyper-parameters search and cross validation.

PySpark seems like a very legit tool but it has its' drawbacks -

No pure python local mode - impossible to use profiler or ipdb
There is latency which is induced by the network architecture (Python driver -> Scala (JVM) -> Python worker)
Traceback is sometimes hard to understand as there is a mix of scala and python errors and log data.

Grisel suggest to use instead Dask and Distributed as a native python packages for parallel and distributed computing.

Being a young project dask also have its' limitation - mainly no distributed shuffles which means it does not support distributed merge, join, groupby and aggregation operations at the moment.

Frontera: open source, large scale web crawling framework (Alexander Sibiryakov, ScrapingHub)

When I read the talk abstract I was not sure what the difference between Frontera and Scrapy. Hearing this talk the goal of Frontera is actually different - schedule crawling and crawling strategy. In their architecture they use scrapy for crawling but can use other crawlers as well.

After introducing Frontera, Sibiryakov showed their result of crawling the Spanish internet and the problems they faced. Some of the solutions were quite trivial - indexing the data differently to avoid hotspot, caching, limiting the depth of crawling \ number of pages per host, etc. I cannot say that I was fully convinced.

Setting up predictive analytics services with Palladium (Andreas Lattner, Otto group)

The evolution of company tool - 80% development time, 20% deployment overhead which tend to repeat between different projects. So why not make a framework and try to automate it. This is exactly the reason Otto group developed Palladium - ease the development, deployment and integration predictive analytics services.

The main limitation of Palladium is that it does no support distributed calculation. Another issue is controlling the results on real time - boosting, filtering, etc. At current time, prediction.io (not written in Python) is in a more mature state.

Spotting trends and tailoring recommendations: PySpark on Big Data in fashion (Martina Pugliese, Mallzee)

Mallzee, as stated by Pugliese is "the tinder of fashion". I.e the applications shows you a stream of fashion items from multiple providers and you swipe them left and right according to your preferences.Their current big challenge is to produce valuable recommendation for each customer.

Their input on one hand are the items and the items information which they crawl from the providers websites and normalize to fit into their set of tags and properties and of course the brand itself. The second input is the user behaviors - swipes, buy actions, favourite and non favourite brands, etc. Their current choice is creating a random forest for each user based on their actions. This approach can work for users with a lot of signals and specifically positive signal (buy and positive swipes). Possibly the next step will be to cluster the users to learn more about the users with log positive signals.

Practical Word2Vec in Gensim (Lev Konstantinovskiy, Gensim)

Gensim is a "topic modeling for humans". It implements many interesting algorithm including lda, word2vec and doc2vec, lot's of potential and interesting to play with.

But this tutorial was a mess.. the class didn't fit to a tutorial, not enough space, not tables people can put their computers on and really run the code.

Konstantinovskiy wanted to present many options and ideas and run too fast between the different algorithms without really explaining them. On the other hand there was not really a focus on running the code and showing the package API. Unfortunately I feel this talked was a bit miss handled.

Bayesian Optimization and it's application to Neural Networks (Moritz Neeb, TU Berlin)

What I would take from this talk is approaching Hyper parameter tuning as an optimization problem. For example when we use grid search for hyper parameter tuning we invest a lot of resources but the experiments are independent from one another and we don't learn from one another. Instead we can possible introduce a method which have some interactions between the different experiments.

Neeb introduced an approach which treat it as a Gaussian Process and at each point try to evaluate the point with the maximal possible gain.

He also mentioned few Python libraries -

Spearmint - designed to automatically run experiments (thus the code name spearmint) in a manner that iteratively adjusts a number of parameters so as to minimize some objective in as few runs as possible.
HPOlib - hyperparameter optimization library
Hyperopt - is a Python library for optimizing over awkward search spaces with real-valued, discrete, and conditional dimensions.

Keynote - Wes McKinney (Cloudera)

Wes McKinney is probably Mr. Open Source he involved and leading some of the most known data science open source projects (in python but not only) - pandas, Apache Parquet, etc.

His talk had roughly 2 parts - talking about the community, code of conduct, etc. and challenges that he believes will play a major role in data science \ python \ big data communities in the near future. Some of the project \ ideas \ challenges he mentioned -

conda-forge
manylinux - Python wheels that work on any linux (almost)

What's new in Deep Learning (Dr Kashif Rasul, Zalando SE)

Deep learning survey on speeds.. Survey of several recent papers in deep learning. Beside on mention of Theano and short example of code no strong connection to Python. I would have compromise to less code and slower pace.

Some of the papers he pointed at -

Understanding the difficulty of training deep feedforward neural networks (Glorot and Bengio) - The term Xavier initialization comes from here. See here for additional information.
Delving Deep into Rectifiers:Surpassing Human-Level Performance on ImageNet Classification (He, Zhang, Ren, Sun - Microsoft research)
Batch Normalization: Accelerating Deep Network Training byReducing Internal Covariate Shift (Ioffe and Szegedy - Google) - A method to accelerate the training of deep neural network by modifying the distribution of activations. See tutorial here.

A full Machine learning pipeline in Scikit-learn vs in scala-Spark: pros and cons (Jose Quesada, Data Science Retreat)

Jose Quesada presented their experience with teaching and using Python Scikit learn versus Scala Spark. The most interesting points in my opinion were the PySpark limitation and future directions and of course comparison of existing features.

One of the most important insights was to use spark DataFrames. spark.ml is built on the top of dataframes and will replace Spark MLlib in the future.

Data Integration in the World of Microservices (Valentine Gogichashvili, Zalando)

Gogichashvili is the head of Data Engineering in Zalando and is working there for already 5.5 years which means he was there during business, structural and technological changes the organization went through.

While he talked about micro-services architecture I think there were two other interesting inputs in the talk -

Team structure - every team is responsible for their own components and infrastructure (including naming) while their align to engineering guidelines such as API structure, data types, etc. which are set by cross team guilds.
Open source code - whenever a team starts a new project by default the project will grow to be an open source unless they reason about why not. Zalando open source projects can be viewed here.

Brand recognition in real-life photos using deep learning (Lukasz Czarnecki, Samsung)

Czarnecki presented his project from Data Science retreat - brand recognition in Instagram photos where he combined neural networks with SVM. He showed the base line he started from (+ some preprocessing) and the steps he made to improve his results.

Some of the steps he did -

Increase training set from 300 images for brand to 800
Multiply the training set by cropping part of the pictures.
Looking on the error and adding training examples to make the NN more general
Increases threshold of the SVM

That's all for now, until next year :)

5 interesting things (16/05/2016)

Mon, 16 May 2016 18:22:12 GMT

Similar Wikipedia Pages - This post present Wikipedia similar pages chrome extension. Phrasing this in other words it is a recommendation system for Wikipedia pages. They are not the first one to do it Wikiwand as well as Wikipedia themselves (in Beta) created such a feature. They compare the result a bit and it would have been interesting to have deeper dive to it. One obvious difference between them and Wikipedia is that while Wikipedia can analyze user behavior and use collaborative filtering or a hybrid approach they only have access to the content data. However they can possibly integrate page view data or edit data to inspect trends and to integrate more temporal data in their recommendations.

https://blog.lateral.io/2016/05/similar-pages-wikipedia/

Cron best practices - while it is a very common to set a cron job it is sometime not fully understood - which user runs the command, where is it run, where to put the scripts, orcehstrating cron jobs and so on. A life hack post.

https://sanctum.geek.nz/arabesque/cron-best-practices/

Resume Dreamer - since the release of TensorFlow as open source many people experience with TensorFlow in different tasks and settings and share their work. This post by untapt team show their efforts on generating CV using TensorFlow. While the task is amusing and one could think of it as a draft to their own CV I lack some technical details about the process. Sawing that the engine recognized the paragraph structure I wonder how it will deal with dates patterns.

https://www.untapt.com/blog/2016/04/25/neural-network-that-dreams-resumes/

Apache Libcloud - Python library for interacting with many of the popular cloud service providers using a unified API. This library have two major use cases as far as I see it - migration between different providers and working with multiple providers on a regular base which I believe is rare. In both cases this package can decrease the "lock-in" to specific provider and that's good thing.

https://libcloud.apache.org/

Graph DataFrames -Databricks expose GraphFrames - graph processing library for Apache Spark. The functionality of Apache Spark is extended step by step and now it is time for Graph algorithms. My guess is that the query syntax will change a bit in the future to support user defined functions and so on but that's a start.

http://graphframes.github.io/user-guide.html

5 interesting things (20/04/2016)

Wed, 20 Apr 2016 09:48:11 GMT

Spoiler detector - Get your annotated data set for free! Nice way to get (though not perfect) an annotated data set and create some social good. Would be interesting to expand it to more movies and television shows and to other social networks. Interesting to see how predictive a model which is trained on one social network preforms on other social networks (for this case and on general) http://www.insightdatascience.com/blog/fanguard.html Deep dive to Python virtualenv - although it only addresses python requirements and not more general system requirements (e.g this case), it is a very important and common tool to make sure that all your python requirements are in place. This post provide some deeper look into virtualenv and pyenv, a python version management tool. https://realpython.com/blog/python/python-virtual-environments-a-primer/ 6 Lesser Known Python Data Analysis Libraries - TL;DR - mrjob, delorean, natsort, tinydb, prettytable and vincent. If I had to write the same blog post I am not sure that those are the packages I would have chosen. mrjob is maintained but I feel it is a bit outdated and there are better way now days to run multi-step mapreduce jobs (e.g Apache Spark). natsort - I understand the need, personally it is easier for me to write the sort function myself and even better avoid sorting as much as possible. prettytable - I prefer pandas printing over this printing. And last but not least - I am not sure that I would really categorize all those packages as "Data Analysis". http://jyotiska.github.io/blog/posts/python_libraries.html Python datacleaner - would definitely be a strong candidate when writing a post such of the above. Designated to make the process of cleaning pandas data frames quicker and easier. Interesting to see how this project will evolve. https://github.com/rhiever/datacleaner Deep Detect - "Open Source + Deep Learning + API + Server". Deep learning version of PredictionIO. It is written in C++11 and uses Caffe for deep learning. It seems natural to me that predictionIO and DeepDetect will cooperate in the future or someone will develop a deep learning template for predictionIO. http://www.deepdetect.com/

Python's Guide to the Galaxy – SPS16

Wed, 23 Mar 2016 17:26:23 GMT

My talk from Swiss Python Summit is online - https://www.youtube.com/watch?v=Q9AU_qETVd8

5 interesting things (22/03/2016) - Hebrew

Tue, 22 Mar 2016 11:51:29 GMT

I usually don't post link to posts in Hebrew because the audience is limited. This time I decided to make Hebrew Special. I appreciate people writing about technology in Hebrew from several reasons - audience and feedback are limited, not all the terms exists in Hebrew, indentation, etc. It took me a while to write this post. To find 5 blogs I can really recommend on. I hope all those blogs will keep being active and some more will join them. This time the links are not to specific posts but to blogs.

Reversim - a podcast for developers but not only for them. Blog \ podcast was founded by Ran Tavori and Ori Lahav and is almost 7 years old now. They also hold an annual summit in Israel. There are several kind of sub-series \ chapter types, my favorite is Bumpers which is a monthly survey of cool things Ran Tavory and friends bumped into that month.

http://www.reversim.com/

Software Archiblog - this blog is written by Lior Bar-On which is currently a Chief Architect @ Gett. On one hand there are deep dive post to AWS services, GO language on so on. On the other hand there are softer posts regarding processes, choosing technology stack, time management and so on.

http://www.softwarearchiblog.com/

the bloggerit - blog by Hadas Sheinfeld, former VP product at ClickTale. This blogs runs now for almost 8 years. The only female writer in this post. Although Reversim host many people from time to time, I don't remember the last time they hosted a female guest. She writes about product, organizational culture as well as on other topics from a wider angle.

http://www.thebloggerit.com/

Navi Sheker - the name of the blog refers to the bible meaning "False prophet" and the subtitle of the blog is "Prophecy was given to fools" which is from Baba Batra one the books in the Mishnah. The posts in this blog mostly analyze twitter chatter and google search in Hebrew, sometimes with correlation to actual events.

https://navisheker.wordpress.com/

I, Code - Comparing to the other blogs this one is relatively young one - few months old. Hopefully it will be active in a year from now (and more of course). The posts in this blog are mostly code samples in Python and CSS. I like the post about how to secure your home in less than 80 lines of code.

http://icode.co.il/

5 interesting things (12/03/2016)

Sat, 12 Mar 2016 17:22:08 GMT

In memory of Udy Brill

pypi analysis - this posts starts with some technical issues, e.g. how to scrape pacakge dependencies from pypi packages and goes on to showing a graph of dependencies between python packages as well as analysis on the graph structure. What we learned from this analysis - requests, django and six are among the most important packages - PageRank and connectivity degree wise. requests also leads in the"betweenness centrality" matrix. While both django and six does not appear in the top 10 results of this matrix. I believe that the reason is the both django and six are used for more specific use-cases \ applications while requests which is more general. One can see that the top-10 packages in "betweenness centrality" matrix are quite general ones (testing and setup) as well as open stack clients.

The post includes two more visualizations - adjacency matrix, which exposes existence of cliques in the graph with some details about them and degree distribution in the graph. A long tail distribution as one can easily expect. Most packages are not imported by anyone else and a few packages are imported by many other packages.

http://kgullikson88.github.io/blog/pypi-analysis.html

Estimate user locations in social media - in the world of targeting and online advertisment one of the challenges is to learn as much as possible about the user for better targeting. Such data includes - gender, age, marital status, field of interest and of course location. No use in advertising a shop which is 500km from the user. But, no one is perfect and so is data. We don't always know user location and this post describe two approaches for estimating user location from social media.

http://www.lab41.org/2-highly-effective-ways-to-estimate-user-location-in-social-media/

10 Lessons from 10 Years of Amazon Web Services - post by Werner Vogels, Amazon CTO concluding 10 years of AWS. Although I sometimes have criticism they (together with other palyer and technology enhancements) truly changed the world. A very interesting read both for AWS users and non users of AWS. http://www.allthingsdistributed.com/2016/03/10-lessons-from-10-years-of-aws.html Introduction to Boosting - TL;DR - think iterative - at each step you improve the model so it will predict well samples which were not predicted correctly so far. Really good overview on the concept of boosting, now I want to use this knowledge and play with it -

https://codesachin.wordpress.com/2016/03/06/a-small-introduction-to-boosting/

curl vs wget - a comparison of the two tools from a contributor to both projects. While (for me) there is not immediate day-to-day implication it is good to look a bit under the hood and to see the pros and cons of each for such a common tools. For me it would be easier to see it as a table.

https://daniel.haxx.se/docs/curl-vs-wget.html

Udy was a friend and a colleague of mine. He died in a track in New Zealand during his honeymoon. For me he had the perfect mix of curiosity, professionalism, team member and a positive person.

I'll remember Udy in everyday moments like eating food with a lot of sauce, reading something about sorting algorithms and hearing Tracy Chapman.

This is a video we made in a company hackathon we worked together. https://www.youtube.com/watch?v=jDZs4Mv_nMU

Spark, EMR, Lambda, AWS

Sat, 27 Feb 2016 15:11:17 GMT

I know the title of this post looks like a collection of buzz words but there is code behind it. AWS Lambda function is a service which allow you to create an action (in this example add an EMR step) according to all kind of events. Such events can be cron expressions or schedule event (once an hour, once a day, etc.), change in S3 files, change in DynamoDB table, etc. The goal of the code is to add an EMR step to an existing EMR cluster. The step can actually be anything- Map Reduce, Spark job, JAR step , etc. This example show how to add a Spark job but it is easy to adjust it to your needs. The parameters for the Spark job can depend on the specific event - for example, put file event data will include data about the specific file. In this example there is only a place holder for the script parameters and the spark configuration parameters. Note - in order to run this Spark job your code should be in Spark master machine. Possible this could be done in the script as well but here we assume it is already in CODE_DIR. https://gist.github.com/tomron/6ebc60cd3450478c7fc4

5 interesting things (25/02/2016)

Thu, 25 Feb 2016 10:26:26 GMT

Caffe On Spark - if it pretty clear that on of the next steps for Spark MLLib is to add deep learning. There are several deep learning frameworks out there and I guess few (if not all of them) will try to integrate with Spark on one way or another and in the end "the fittest will survive", i.e. will be officially supported by Spark. What will that be? my guess is an Apache project, maybe SINGA, maybe a popular project that will become an Apache project. However, other extensions will continue to exist for specific use-cases and for users who prefer specific frameworks (e.g. reusing already existing code). https://github.com/yahoo/CaffeOnSpark Kibana plugins basics - I discovered this post while looking for a way to add a reasonable KPI visualization for elasticsearch. This is actually a series of 4 posts going over how to write and deploy (i.e. register) a custom Kibana plugin. Unfortunately we use AWS Elasticsearch service which does not (currently) support customized plugins and therefore we suspend it for now.

https://www.timroes.de/2015/12/02/writing-kibana-4-plugins-basics/

Bonus track - creating traffic light visualization - http://logz.io/blog/kibana-visualizations/

Thoughtsworks radar - every few months Thoughtsworks publish their technology rating. There are 4 ratings - adopt, trail, asses and hold. The products / tools / methodology are divided to 4 groups - techinques, tools, platforms and languages and frameworks. It is specially interesting to see how ranks change over time. I found database products to be a little under represented. I guess there is also a bit of bias towards tools and companies they work with but nonetheless it is a interesting read.

https://www.thoughtworks.com/radar

. files - dot files are a common way to set all kind of configurations from .gitignore to .bashrc and everything in the middle. You probably have your configuration that you like and copy it from project to project. You still might find this repository helpful as it collects common configuration settings -

https://github.com/holman/dotfiles

Bonus track - git ignore by programming language

https://github.com/github/gitignore

SQL tabs - simple but get the job done. Yet another visualization tool above postgresql. Super simple installation and start up (on mac os environment) even for non technical people. http://www.sqltabs.com/

Why not AWS Elasticsearch service

Fri, 19 Feb 2016 22:53:53 GMT

Almost 5 month ago AWS exposed a new service - “Elasticsearch service”. So we can now have a managed elasticsearch cluster and we can have Kibana on it without the hassle of managing it. But then we need to work with it and discover the bleeding edges -

Old version - current Elasticsearch version is 2.2.0 and 1.7.* is maintained. Elasticsearch version in AWS is 1.5.2. It is 8 months old version, it is ages in this industry. Kibana current version is 4.4.0 while Kibana version offered on AWS is 4.0.3. Many enhancements, bug fixes, breaking changes and AWS is behind. This is specially interesting comparing to the recent velocity on EMR images. As there are some breaking changes between 1.* versions and 2.* I would at least expected the option to choose a version. Even more annoying there is currently no data about when and how AWS going to upgrade the versions.
Plugins not supported - adding plugins is relevant for expanding Elasticsearch usability, monitoring and for adding customized Kibana visualisations (e.g for KPI's). I believe every company which uses Elasticsearch will have this need in some point. AWS currently support only a close set of plugins and don’t enable installing your own plugin and there is no data about future plans.
Scripting not supported - scripting is relevant for adding data, processing data, interactions between fields, etc. Again a need that I believes that raises in every company after a while. This features also is not enabled in AWS Elasticsearch Service. There are ways to by-pass some of it but why work hard if Elastic already created this functionality? Here too, not clear if there are any future plans to enable it.
No VPC support - this means that the Elasticsearch endpoint is publicly-accessible. There are few solutions but it is a security issues and more over it does not comply with the other AWS products logic.
Backup issues - no easy way to export data and to control how frequent and when your data is backed up. AWS cloud plugin for Elasticsearch can solved it, but it is not supported..

Conclusion - Elasticsearch is a very strong tool and even stronger when combined with Kibana. On the other hand, AWS usually produce very good products.But this time, the combination of those two is not there yet. It seems to me like they might jumped to the water too early (possibly in order to compete with Found \ Elastic Cloud) with a product which is not mature enough. Most of the services AWS offer are based on their own innovation and \ or development and are not coupled so strong with external product (the other product I can think of is elasticcache which I have not worked with yet). From a business point of view, it is weird for me to see such a dependency on external product. If there is a high demand for Elasticsearch on AWS I would prefer to have a tutorial explaining how to set a cluster rather than a doubtful product.

5 interesting things (13/02/2016)

Sat, 13 Feb 2016 19:22:22 GMT

Ads @ stackoverflow - Steve from Stackoverflow ad managment team explains their view about ads on the site. This is a very user centric approach along with strong branding and differentiating themselves. This approach is not scalable for a sites with many topics and diverse audience but it fits their niche.

https://blog.stackoverflow.com/2016/02/why-stack-overflow-doesnt-care-about-ad-blockers/

Serverless Python web applications - taking scalability and cost effective to the next level. Zappa is a Python web applications where the server is created using AWS Lambda function after an HTTP request came through AWS API Gateway. The python web framework under it is currently Django but it should be customizable to also support Flask, pylons, etc.

https://gun.io/blog/announcing-zappa-serverless-python-aws-lambda/

Slack @ Home - the interesting thing for me is the transformation between business tools and free time tools \ technologies. Can we use the knowledge, tools, methods we use in work at home and the other way around?

The first feature for me would be shopping - collecting easily the list of things I need to get in the supermarket, drug store, etc and updating it back to the slack. Scheduling is also interesting but there a google calendar integration so I think it is pretty easy. And maybe integration with WhatsApp possibly for image backup and sharing. Of course, it does not necessary have to be slack, any other equivalent, preferred open source tool is good as well.

http://labs.earthpeople.se/2016/02/my-family-uses-slack/

Towards an Open Banking API Standard - one can probably not underestimate this move. I believe that an open banking API will lead many technology driven progress in the personal banking. For example - aggregating data from several account, comparing fees, etc. It will be interesting to see how the banks will adopt this standard as banks are heavily lead by regulations and are slow moving organizations by their nature. This specific effort is directed to UK banking systems but it will be interesting to follow if additional financial organization in and outside of UK will adopt it as well.

http://jackgavigan.com/2016/02/10/towards-an-open-banking-api-standard/

3.57 Degrees of separation - as a matter of fact I find the idea ok six degrees of separation romantic and not very interesting. In the edge of social networks and big data we can actually check it, or evaluate it. This post by Facebook research tries to estimate it. The interesting part for me in this post was the mathematical background and the engineering effort of scaling it. Please note that data is mostly available for first world countries and therefore although it is an estimation it probably need another factor of distance.

https://research.facebook.com/blog/three-and-a-half-degrees-of-separation/

Cassandra Day Berlin - 8 things I learnt about Apache Cassandra

Thu, 11 Feb 2016 19:09:38 GMT

Since I didn't know anything about Apache Cassandra (beside being a distributed database) I could probably replace the 8 in the title to any number.. Here is a summary of few things I learned in Cassandra Day today.

What is Apache Cassandra - open source distributed database donated to Apache foundation by Facebook (actually first to google code and then to ASF). It is decentralized - meaning all nodes are born equal and there are no masters or slaves. It is schema-full database see here more about the Myth of Schema-less databases. It supports replications, i.e. data redundancy with default replication factor of 3 and multiple data centers (both physical and virtual). and you can control the consistency level (see next). Therefore it is AP in the CAP theorem. It uses CQL - Cassandra Query Language.
Controlling consistency - having the data replicated in several nodes one can read and write data in 3 ways. The higher consistency level the longer the latency is.
- One - reading or writing to one node is enough.
- Quorum - i.e. value was written \ retrieved at least from half of the relevant nodes. Latest win - when retrieving the data from several nodes the value with the latest time-stamp counts.
- All - need to write \ retrieve data from all the relevant nodes. T Note that all mode is dangerous since it cancels the high available. If one of the replicas is not available we will get no answer.
Data Modelling is important - Isn't it right to every database? Well yes. But the issue here is again the trade off. One on hand I feel that data modelling is sometimes neglected on NoSQL databases since we can just through data there and scale it. On the other hand, due to some limitations (see CQL next..) data modelling for Cassandra is quite opposite to best practices in RDBMS. Know your queries in advance and build the schema (e.g keyspace) accordingly.
CQL - Cassandra Query Language. At the first talk they said "Yes, it is exactly like SQL". Well, not exactly. First - no joins, this influences dramatically about the data modelling. Limited aggregations functionality. Update command always work even if record does not exist (can be controlled). See here more about CQL vs SQL.
CQL containers - there are 3 types of containers
- Set - a container of items sorted by the type compare operand.
- List - a container of items sorted by the insertion operand
- Map - a key, value container sorted by the type compare operand of the keys. This is kind of a hack which allow you to have it a bit schema less. Up to 64k items in a map.
Spark connector - connecting to one of the most trending technologies.
Solr integration - Cassandra is not a document database. However, sometime users how choose Cassandra as their main solution have indexing \ search needs. Datastax have a patch which also allow you to search the data in RAM (data which was not yet written to the Lucene indices).
How to go on from here? How can I learn more -
- DataStax academy
- Official documentation - by version
- Stackoverflow
- Planet Cassandra
- Al's Cassandra 2.1 tuning guide

There are much more for me to learn about Apache Cassandra and more things I learned in this day but this is a short review.

Swiss Python Summit 2016

Mon, 08 Feb 2016 23:06:33 GMT

I participated in Swiss Python Summit today, where I gave a talk about "Python's guide to the Galaxy". Code and slides are available in my GitHub. All the talk were recorded and they will probably be online within a few hours \ days.

Beside the best signs I can think of the program was great and very diverse in the topics - music, 3d graphics, astrophysics and so on. Even the order of the program was very good - super cool talks after lunch to keep everyone a wake. The speakers were also relatively diverse - 4 out of 9 speaker were women, different backgrounds, etc. Unfortunately there were only few women in the crowd.

I was very lucky to give the first talk so I could then concentrate on the other talks. One very strong point of the conference was the diversity of the talks.

The second was "API Design is Hard" by Dave Halter, creator and maintainer of Jedi. It was more conceptional talk then a technical one which was enriched by lessons he learned creating and maintaining Jedi. I believe this is an important talk regardless the specific programming language.

Next session opened with Armin Rigo talking about CFFI - "C Foreign Function Interface and PyPy interrupter. Interesting datum - CFFI is currently ranked 23rd by PyPI Ranking above Django and aws cli. Probably thank to the fact that it is integrated in Cryptography package and others. Pypy - is a python interpreter written in Python (rather then in C comparing to the most common implementation) and is designed for speed optimization.

Martin Christen had 3 parts to his talk. First he talked about 3d modeling with Blender, then he introduced some more low levels possibilities for 3d modeling and he ended up introducing some of the many open source projects he works on. Those projects mostly involve open 3d maps and globe maps.

After lunch came Matthieu Amiguet, co-founder of Les Chemins de Travers showed how you can manipulate sound using the relevant packages. This was really outstanding and non standard talk. One of the packages he currently uses is pyo.

Next talk was given by Chihway Chang from ETH Zürich who spoke about " Coding / Decoding the Cosmos - Python application in Astrophysics". As she said, and I fully agree - "Python is everywhere in science today". She talked about the found usage of Python in her department, both in Astrophysics dedicated packages (Astropysics) and using more general purposes packages (SciPy, Numpy, pandas) and packages they write themselves. She presented 2 issues they work on - mapping dark matter and calibrating radio telescope with drone. I strongly suggest to see at least a second part of this talk to original scientific thinking.

Michael Rüegg demonstrated a Scrapy pipeline which start in crawling running races results and ends up in elastic search. This was probably the most technical talk of the day and the concrete project was of additional interest for me.

Jacinda Shelly showed us how magic is done, live coding. Specifically speaking about magic functions in IPython and other cool features.

Last but not least, Florian Bruhin speaking about why and how we should use, how we can expand the basic functionality and so on. In addition he also spoke about other projects he is involved in: qutebrowser - a python browser and tox - a generic virtualenv management and test command line tool.

This is the first edition of the conference and for me it was super interesting and I really loved the mixture of the talks. Hope to be there again next year.

5 interesting things (24/01/2015)

Sun, 24 Jan 2016 12:33:18 GMT

Top 30 books ranked by total number of links to Amazon in Hacker News comments - I must say this analysis surprises me. First because of the low numbers - most common book was mentioned only 53 times. Second, I expected "The Lean Startup" to be one of the 30 top books. I guess one of the caveats as described in the post itself are the links themselves that may differ a bit or change over the years and therefore hard to match. http://ramiro.org/vis/hn-most-linked-books/

LDA2VEC - getting the best from both worlds LDA + word2vec. Stitch fix definitely brand themselves as one of the leading companies technology and research wise doing some very interesting things. http://www.slideshare.net/ChristopherMoody3/word2vec-lda-and-introducing-a-new-hybrid-algorithm-lda2vec-57135994

Automatic colorizing - coloring gray scale images using deep learning. There are many works now trying and checking tensor flow particularly and deep learning on general. I really liked this case, maybe also because it uses a set which was already trained for other purposes.

http://tinyclouds.org/colorize/

Gestures Typing - First of all I liked the question - answer format of the post. The post looks for an optimal keyboard. But what is optimal keyboard? depends who you ask. On the other hand I think qwerty (or qwertz in Germany) is such a strong standard that I wonder what will make it change.

http://nbviewer.jupyter.org/url/norvig.com/ipython/Gesture%20Typing.ipynb

Roomba + voice recognition - problems of the very rich part of the world. Yesterday night we spoke with friends which complained their Roomba sometimes get stuck in all kind of corners, cable and so on in their house. We talked about several ideas to solve it, one was voice command and this morning I found a guy which already tried it -

http://makezine.com/projects/use-raspberry-pi-for-voice-control/

5 interesting things (10/01/2016)

Sun, 10 Jan 2016 10:23:48 GMT

Everything.me open source their inheritance - Everything.me was an Israeli startup closed few weeks ago. They now open source major parts of their code and tools including their prediction algorithm, Re:dash and others. https://medium.com/@joeysim/everythingme-open-and-out-6ed94b436e4c Python 3 module of the week - over the years Python Module of the week by Doug Hellman was one of the most reliable documentation resources for Python's standard library. The documentation is always accompanied with very good examples to almost any functionality. It is also available as a command line tool and was translated to Chinese, German, Italian, Spanish and Japanese. And.. it is now updated to Python version 3.5. Kudos. https://pymotw.com/3/ The Star wars social network - although I'm not a star wars fan I think this provides a very accessible introduction to graph algorithms and measurements. http://evelinag.com/blog/2015/12-15-star-wars-social-network/ D3 in Jupyter - an intersection between 2 tools which I use quite a lot and find very important for data scientist. Not those tools specifically but tools which make the data science magic more approachable to others so one can share its' findings and get feedback. http://multithreaded.stitchfix.com/blog/2015/12/15/d3-jupyter/ Bonus - Building interactive dashboards with Jupyter - http://blog.dominodatalab.com/interactive-dashboards-in-jupyter/ Tl;dr man page - given a bash command creates a tldr of the man page. More of a gimmick but a nice one. http://www.ostera.io/tldr.jsx/

Surviving Black Friday & Turning Behavioural Signals Into User Profiles

Thu, 07 Jan 2016 15:39:15 GMT

I was on a visit in Israel and went to "Surviving Black Friday & Turning Behavioural Signals Into User Profiles" meetup. It is a very long name which actually implying on 2 talks. The meetup took palce in Sears Israel offices, which is the department behind Shop Your Way.

The first talk - "Surviving Black Friday" was given by Omri Fima who is a resilience tech lead in Sears. Omri talked about resilience and scalability lessons learned based on Black Friday. Every internet shopping site knows that the traffic is much higher on Black Friday, so how can you prepare and test if your system can deal with such a load? What happens if one service fails? What is a graceful failure and what is less graceful? He presented 7 steps to make your service more stable and mentioned few tools both for testing and development.

Pablo Rosenman, VP Development @ Adience gave the second talk - "Turning Behavioural Signals Into User Profiles" (slides, video). Pablo presented 2 of Adience products - Adience SDK and Events SDK and showed how they use AWS services in their pipeline. He talked about Adience pipeline and what were their main concerns and focus when designing and implementing it - scability, decoupling and cost effective. In the end he also presented what they would do differently if they would design it today.

All in all, interesting talks and a very good atmosphere. Looking forward that Omri's slide will also be available online.

5 interesting things (13/12/2015)

Sun, 13 Dec 2015 13:09:39 GMT

Systemml - distributed and declarative machine learning platform. Looks like a promising project which now joined Apache Software foundation and was initially developed by IBM. I wonder how it will influence the development of Spark MLLib. http://systemml.apache.org/ Monopoly as Markov Chain - I guess I am developing a fetish to Markov chains. Although this model is not always realistic (so as in this case..) it is amazing what we can get out of it and the approximate simulations we can create. This post simulates a Monopoly game using Markov chains and does a very interesting job. Monopoly is linear in the sense that you must move according to the dice and the choices you make are limited (buy or don't buy). In contrast to backgammon where you also have strategy involved and therefore it is a better choice to model it with Markov chains. http://koaning.io/monopoly-simulations.html

Vocabulary - "Python Module to get Meanings, Synonyms and what not for a given word". This module brands itself as an alternative to NLTK presenting data about meaning, synonyms, antonyms, part of speech, pronunciation, etc with a leaner approach and more pythonic approach. I don't know if is as good and NLTK or will evolve there but it sure looks like an alternative worth checking. http://vocabulary.readthedocs.org/en/latest/ Probability recap - If you forgot probability class from university this will probably be a good recap. However, if you work as a data scientist you probably used those daily. But code visualization, good examples and good way to share your knowledge with other colleagues. http://blog.cloudera.com/blog/2015/12/common-probability-distributions-the-data-scientists-crib-sheet/ What we talk about when we talk about distributed systems - for once a non misleading title. Thinking about the distributed systems course I took on grad school this would have been a great introduction. http://videlalvaro.github.io/2015/12/learning-about-distributed-systems.html

5 interesting things (1/12/2015)

Tue, 01 Dec 2015 13:22:54 GMT

Improving My CLI's Autocomplete with Markov Chains - Markov chains are the basis for many auto-complete algorithms we know and use on daily basis, e.g keyboards on mobile devices. In this case it is a developer hack to improve auto-complete in a development tool. It is always nice when theory comes to life. http://nicolewhite.github.io/2015/10/05/improving-cycli-autocomplete-markov-chains.html 10 more lessons learned from building Machine Learning systems - slides of a presentation by Xavier Amatriain, VP Engineering at Quora (previously Director Algorithms Engineering @Netflix). Very insightful presentation (I would of loved to hear the full one). The name refers to a lecture by Amatriain called "10 lessons learned from building Machine Learning systems" exactly a year before). http://www.slideshare.net/xamat/10-more-lessons-learned-from-building-machine-learning-systems http://www.slideshare.net/xamat/10-lessons-learned-from-building-machine-learning-systems

Beyond One-Hot: an exploration of categorical variables - It is not all about numbers.. in many cases features are not numeric. If we are lucky - features will be binary - has \ does not have a symptom, spam \ not spam or ordinal - amount of pain a patient experiences, etc. But sometimes it is neither - e.g a state, color, etc. What then? this post compare several techniques to deal with categorical variables. While it is very basic it well explained (although examples would have helped) and it can give a great intuition for someone who faces those problems for the first time. http://willmcginnis.com/2015/11/29/beyond-one-hot-an-exploration-of-categorical-variables/

Bandit Algorithms for Bullying - Getting More Lunch Money - this post explains bandit algorithm using very common, easy to understand example. However, I would build the post a bit different. There is a lot of text telling the story comparing to the scientific parts. In my opinion the scientific parts should be emphasized a bit more (bold text, bullets, etc.) http://1oclockbuzz.com/2015/11/24/bandit-algorithms-for-bullying-getting-more-lunch-money/

C.H.I.P vs Pi zero - The new Pi zero made a lot of buzz in the last week offering a computer for 5$. However, Pi zero is not the only player in the field of sub 10$ computers. This post compares Pi zero and C.H.I.P spec and abilities. There are many comments saying the comparison is biased towards C.H.I.P (ignoring shipping costs, unfair comparison of cable costs, reputation, etc.) but overall I think it is worth reading. http://makezine.com/2015/11/28/chip-vs-pi-zero/

KNN approximation Apache Spark

Thu, 19 Nov 2015 19:30:31 GMT

K-nearest-neighbors is a very well known classification algorithm. It is based on the phrase - "show me who your friends are and I'll tell you who you are".

Apache Spark MLLib contains several algorithms including linear regression, k-means, etc . But it does not currently include an implementation to KNN. One of the reasons for that is the time complexity it requires (roughly n^2 where n is the number of items, ignoring the dimension). At Apache Spark JIRA you can see 2 tickets involving this issue - SPARK-2335, SPARK-2336. The first ask for KNN feature and discuss the difficulties. The second, open based on the first, discuss approximations to KNN and wish to implement it.

I implemented a very naive approximation to KNN algorithm on Apache Spark with a distance function of similarity (looking for max) rather hen euclidian but this can be easily changed (change distance function and change sorting key in line 71).

The algorithm is based on splitting the data to partitions and calculating item distances only in the same partition. You can increase the accuracy either by decreasing the number of partitions (compare to more items) or by repeating the process several times (repartition differently every time) and choose the best results.

This calculation retrieves the list of the most similar neighbors and then one can decide how to use this data.

My gist - https://gist.github.com/tomron/70e5fefe128214b7d2a1

5 interesting things (15/11/2015)

Sun, 15 Nov 2015 15:15:25 GMT

Counting things in Python - This post spotlights very nicely and simple how Python and Pythonic writing changed over the years. One of the interesting things in this post is the analogy to natural languages. Natural languages also changes and evolve over time - slang, new phrases, out-dates expressions etc. and apparently so is programming languages.

http://treyhunner.com/2015/11/counting-things-in-python/

Travelling Beer drinker problem - Figuring out the shortest road trip path to visit best microbreweries in US. Finally a good use to the travelling sales man problem :). Data science wise there is not much into this problem - connecting the dots between location data and google API but that is an exciting ground for this connection. http://flowingdata.com/2015/10/26/top-brewery-road-trip-routed-algorithmically/ Hacker news 9 year statistics - analysis about Hacker news activities, volume, users and trends over the last 9 years. Interesting specially because it became such a central place to consume technology news.

http://debarghyadas.com/writes/looking-back-at-9-years-of-hacker-news/

Evolving strategies for an iterated Prisoner's Dilemma tournament - I don't get to read many posts about evolutionary algorithms or practical game theory. So such in in-depth post with python implementation is really refreshing.

http://mojones.net/evolving-strategies-for-an-iterated-prisoners-dilemma-tournament.html

Visualizing Chess data with ggplot - Although I'm not an R user I love chess (and I'm horrible player) and I loved the analysis

http://jkunst.com/r/visualizing-chess-data-with-ggplot/

Five interesting things (25/10/2015)

Sun, 25 Oct 2015 19:19:18 GMT

Whatsapp CLI - control you server using Whatsapp. The next step for me is to connect my home (heating, lights, doors) and control it via Whatsapp. To be honest I assume that some such implementations exists. https://github.com/KarimJedda/whatsappcli

Receipt parser - A project which was created as part of Trivago Hackathon. I actually thought about the need of this product many times - first for personal accounting, keep an eye on my spending. Second for taking it to the next steps - alerts about things I should buy, alerts for buying things I don't need and alerts for cheaper prices.

http://tech.trivago.com/2015/10/06/python_receipt_parser/ Time magazine visual trends - good ideas are priceless and when the implementation is also clean, nice and reveals interesting insights it is even more exciting. http://www.pyimagesearch.com/2015/10/19/analyzing-91-years-of-time-magazine-covers-for-visual-trends Google spreadsheet to ElasticSearch - to tell the truth, I could not think of a worse architecture than to use Google spreadsheet as a database (or a proxy in the way to a DB). Having said that, Elastic release a google spreadsheet plugin to import spreadsheet content to ElasticSearch instance. https://www.elastic.co/blog/introducing-google-sheets-to-elasticsearch-add-on Snakefooding python - snakefood is a tool to create graph dependencies. This post show the dependency graphs for some very common python libraries (Flask, django, Celery, requests, etc.). It presents the pros and cons of using snakefood, what it exposes and what it does not expose (many small files -> many imports -> complex dependency graph vs one file, spaghetti code -> no imports -> very clear simple graph). I find it as a tool that supports developing and detecting non necessary dependencies. http://grokcode.com/864/snakefooding-python-code-for-complexity-visualization/

AWS loft Berlin

Fri, 16 Oct 2015 18:23:49 GMT

This week Amazon opened a loft in Berlin which is suppose to be open for 4 weeks. The loft is currently on pilot and there are several other lofts around the world. I think it is a very good strategic call to have it in Berlin as the startup is a emerging and many people want to try and learn more about cloud services while the hands-on experience is sometimes limited.

So what is going on in the loft? It is open everyday 10-18 and there is amazon employee "in duty" which you can consult with regarding AWS services. A very inviting work space. And workshops, demos, bootcamp, etc. All, of course, related to AWS services.

I took part in two workshops on Thursday morning - "An overview of Hadoop & Spark, using Amazon Elastic MapReduce" and "Processing streams of data with Amazon Kinesis (and other tools)". Both lectures were given by Michael Hanisch, solution architect at Amazon. The first talk was a bit messy as it covered many topics but eventually ended up jumping here and there between general things about Hadoop, tips about EMR and changes in the AMI concepts and versioning and clues about Spark.

The second talk was much more focused. It started by introducing the need to Amazon Kinesis. Then explaining the architecture - producer, streams, shards, clients, clearing up the capabilities and constraints also mentioning kinesis autoscaling utils.. The next step was a deeper dive to kinesis producer library and kinesis client library. Moving forward to kinesis firehose (which was introduced in the re:invent last week) and integration with additional input and output sources and aws services. To sum up the talk ended with tips and best practices. AWS Lambda was also mentioned several times over the talk as a tool to process stream data.

Quite exciting time to be in Berlin.

5 interesting things (06/10/2015)

Tue, 06 Oct 2015 22:18:29 GMT

Restaurants recommendations - I read quite a lot about recommendation systems lately and I love this post because it talks about a restaurant domain while many of the posts related to recommendation system refers to music, television and movies. And geospatial features are very important here comparing to movie, music and television recommendations. http://www.slideshare.net/SudeepDasPhD/recsys-2015-making-meaningful-restaurant-recommendations-at-opentable Data Science workflow - this blog post present a well structured approach to data science process. While I loved the structured approach the entry point - "get data, have fun with it" is twisted. I believe that when working on a product most of the time you will want to solve a problem or introduce a new feature, i.e. you already have the question you want to answer rather than explore a dataset and think about the questions you can answer with it. Also missed part of documenting you work. http://blog.binaryedge.io/2015/09/08/the-data-science-workflow/ AWS in plain English - or AWS for humans. If you are not that yet or not familiar with the different services this is a nice way to introduce the terminology. https://www.expeditedssl.com/aws-in-plain-english Fedrer vs Djokovic - why not Nadal? it seem kind of abuse in the product but for sure it exposes the product and make some buzz. https://www.elastic.co/blog/building-dashboards-using-data-from-the-federer-&-djokovic-tennis-rivalry What to do with small data - big data is one of the buzz words in the last few years. While it was previously whispered only by tech people it is now a common, well known phrase. But, many companies do not really have big data, they have few users and need to perform well for those users and one day they might have big data, many users and tons of features. Until then, there are some clues in this post. https://medium.com/rants-on-machine-learning/what-to-do-with-small-data-d253254d1a89

#dmconf15

Sat, 19 Sep 2015 20:13:06 GMT

I got a free ticket to distributed matters conference winning the challenge of the python big data analytics meetup group. The line up was impressive (including Kyle Kingsbury from project "call me maybe"), the topics were very interesting and the location was very nice (first time I hear a talk in a disco party hall) and close to my house, perfect match. Due to earlier commitments I could only stay half of the day but what a day :)

Starting with Kyle Kingsbury (Stripe) as a key note talking about "call me maybe" and the jepsen test. The basic setup of the test is creating a cluster of some database, injecting random events (read, write, update, query, etc) and verify the database behavior. Next level include injecting all kind of failures - network errors, node pause, etc. Following the description of the test Kingsbury gave a brief summary of the results. Project "call me maybe" result creates pressure on the different database vendors to improve their products and align with the promises they make. It definitely serve as the watch dog of this industry. It is sometime amazing how not tested some products are.

The sentence I took with me from his talk is the choosing a database is a contract and you should check all the relevant aspects of it. Settings, complexity and needs sometimes change over time but as long as the initial contract is honored no complains.

The second talk I went to was Michael Hackstein's (ArangoDB) talk "NoSQL meets microservices". Hackstein presented a problem of multiple calls to multiple databases where each answer different need and presented the solution - multi-model NoSQL database. Single call can answer all needs. ArangoDB is an example of such a database, OrientDB is also a major competitor. The problem multi-model database might be phrased as just multi-model data. I.e. documents data, graph data, key-value data, etc (see footnote 1). The rest of the talk focused on ArangoDB different features -

foxx - REST API over the database which works as a microservice.
drivers - including python, php, java, js, ruby, etc.
AQL - ArrangoDB query language

Next talk was given by Charity Majors (Parse - acquired by Facebook) as she introduced herself - "hate software for living". This talk was really really great one. Majors is super experienced, worked with many technologies and very charismatic. She spoke about her many years experience about upgrading databases. Sound easy? well.. not that much. Stepping a step back, why would you want to upgrade a database - new feature, improved preformance, bug fixing, etc.. So what can go wrong? well, many things. From down time, losing data, losing transactions, different hashing and indexing, optimization which are not optimized to your data, changes in clients, and much more fun. Good news - you have enough interesting and important data to care for.

What should you do about it? tl;dr - be paranoid. Longer version -

Change only one component at a time
read the docs, change logs, release notes (also mentioned by Kyle Kingsbury)
Don't believe to database vendors benchmarks, test it for your expected load (if you can expected it, another painful point) - at least 24h data, make sure to clear the cache.
Take consistent snapshots
Prepare for rollback option.

I really hope her slides will be published soon since it was a very informative talk.

Tour de Force: Stream Based Textanalysis by Hendrik Saly, Stefan Siprell (codecentric). This talk was classified as expert level and I expected it to a bit more then what it actually was. Many buzzwords were thrown around - R, machine learning, twitter, Apache spark, elasticsearch, kibana, aws, etc. but the big picture, how those technologies relate to each, what are the alternatives. For example, for the goal of enriching the data R can be easily replaced with python. Or in order to read from twitter why use files and not use Spark streaming capabilities (or some queue). Beside Kibana being a great and easy to use tool, preferring Elasticsearch over other options was not clear. To sum up - I also missed some reference to the distributed properties of those tools. This is the reason we were all there after all, not?

Footnotes

1 Multi-model data is just a way of thinking of about data structures in patterns we are used to. Maybe there are other more efficient way to think of this data?

2. Sponsor wise - on one hand there were few sponsors that didn't give any talk, for example idealo. On the other hand there were few companies I expected to sponsor the conference, i.e crate.io, Elastic, mongodb which was mentioned a lot but unfortunately didn't give any dedicated talk.

3. Database vendors business models. I think that the most popular model as of today is to develop a database, possibly oen source (elasticsearch, ArangoDb etc.) and make money from supporting it. Is there any other way to make money from database vendoring those days (hosting is not included under vendoring).

4. Diversity in tech - I saw no more than 5 women in the crowd.

5 interesting things (11/9/2015)

Fri, 11 Sep 2015 12:28:40 GMT

Density based clustering - the clearest and most practical guide I read about density based clustering. http://blog.dominodatalab.com/topology-and-density-based-clustering/ Word segment - this python library which is train with over a trillion-word corpus aims to help segment text to words. E.g "thisisatest" to "this is a test". I tried a random example -"helloworld' and it didn't split it at all. I tried other examples as well ("mynameis<x>", "ilivein<y>", etc) and it worked well. Beside the segmentation functionality it also offers unigrams and bigrams counting this can be usable for all kind of applications without the need to get the data, clean it and process it yourself. Numbers do not appear in the unigram count, I find it interesting for other needs rather than splitting. http://www.grantjenks.com/docs/wordsegment Funny haha?! Predicting if a joke is funny or not based on the words it contains using Naive Bayes classifier from NLTK package. It is a good and funny beginners tutorial. NLTK contains additional classifiers beside Naive Bayes Classifier, e..g Decision Tree Classifier, it would also be interesting to see how they preform on this dataset. http://vknight.org/unpeudemath/code/2015/06/14/natural-language-and-predicting-funny/ Scaling decision forests lessons learned - some of those lessons are specific to decision forests, some are for scaling and some are general good practices. I wish there was a more found discussion about dealing with missing data \ features. Not specifically in this post but in general. http://blog.siftscience.com/blog/2015/large-scale-decision-forests-lessons-learned Redshift howto - very extensive guide to redshift, mostly admin and configuration related stuff. I miss another chapter regarding tools above redshift such as re:dash. https://www.periscope.io/amazon-redshift-guide

5 interesting things (05/09/2015)

Sat, 05 Sep 2015 13:22:00 GMT

Time map visualization of discrete event - Very good idea for visualization of discrete events when the order of events is not important but rather the general pattern. Good to visualize time between failures, time between visits of users in site \ user actions, etc. https://districtdatalabs.silvrback.com/time-maps-visualizing-discrete-events-across-many-timescales Cyber attacks map - so cool http://map.norsecorp.com/ Why different people think different - the actual title of this post is "Why a Mathematician, Statistician, & Machine Learner Solve the Same Problem Differently" but I think it misses in many aspects. First, the comparison is a bit shallow ignoring non parametric statistics, machine learning models with hyper-paremeters, etc. Machine does make assumptions on the data - by choosing the features you use (even if they'll eventually assigned with weight of 0) you make assumption about the data. Moreover, choosing the model, kernel etc. assumes something regarding the features' distribution. Researchers, data scientist, statisticians, people think differently. Some of them tend to use tools they know and worked for them before, some of them want to use new tools and ideas. I believe the people with same education (ML, statisticians, etc) but from different sources \ institutes will also have different approaches no matching the theorem in this post. http://www.galvanize.com/blog/2015/08/26/why-a-mathematician-statistician-machine-learner-solve-the-same-problem-differently-2 Spreadsheets are graphs? I like this post because it presents some fresh spirit to a painful problem we all experience - sharing documents \ [reserving knowledge. In almost every organizations there are all kind of documents (spreadsheets, word documents, presentations, RFPs, etc.) but it is almost never connected among them and almost always a mess. So this is another angle too look on this problem. http://neo4j.com/blog/spreadsheets-are-graphs-too/ Cross validation done wrong - things that are clear when thinking about them but we don't usually spend time thinking on them. The bottom line is always isolate completely between your training set to the cross validation and test sets. http://www.alfredo.motta.name/cross-validation-done-wrong/

Apache Flink workshop

Fri, 04 Sep 2015 13:25:51 GMT

On Wednesday I took part in "Stream Processing with Apache Flink". The workshop was hosted by Carmeq and was super generous.

Apache Flink is a distributed streaming dataflow engine. There are several obvious competitors including Apache Spark, Apache storm and MapReduce (and possible apache tez).

The main question for me when coming to adopt a new tool is why it is better than what I already use, which problems that it solves for me.

Apache Flink's main advantages comparing to Apache Storm is the batch capabilities, support windowing and exactly once guarantee. Apache storm is designed for event processing, i.e. streaming data. The streaming window allow very easy and native aggregation by both time and capacity windows.

Advantages comparing to MapReduce are strong support of pipelines and iterative jobs as well as many types of data - Flink is more high level than MR. And of course the streaming.

Comparing to Apache Spark, the implementation of spark streaming is different and is implemented as small batches. Apache Spark is limited by memory size which Flink is less sensitive to it. However, I think Spark has a very big advantage at the moment by having API's to R and Python (in addition to Scala and Java) which are very common for data scientist while Flink currently supports only Scala and Java.

Both Spark and Flink has Graph (Graphx and Gelly) and machine learning (MLLib and FlinkML) support which make them much more friendlier and high level than both MapReduce and Storm.

I think both Spark and Flink have a lot of things in common and knowing one it is relatively easy to switch to the other. Currently Apache spark is much more popular - 2273 results vs 59 results on stackoverflow and 8270000 results vs 363000 on google.

For further reading - flink overview.

The workshop focused on flink and we went through the slides and exercises in the flink training site. There were few issues - bugs, java version, flink version issues but it was generally well organized and the guides were eager to help and to explain.

5 interesting things (26/08/2015)

Wed, 26 Aug 2015 17:12:00 GMT

Best programming language ever!!!!!1 - I really agree with this guy approach and the way he presents it. No better tool but better choices. http://coding-geek.com/the-best-programming-language/ Apache Spark @ TripAdvisor - the title of this post is "Using Apache Spark for Massively Parallel NLP" but this is a bit misleading. Apache Spark is discussed only in the last third of the post and does not dive to the technical details as I expected or how Spark integrates with the rest of their architecture. However, the methodology they present regarding asking the users questions and the inherent users' bias he points at. I'll wait for the next post where he says he will talk more about the algorithm. http://engineering.tripadvisor.com/using-apache-spark-for-massively-parallel-nlp/ Let me guess where are you from? I love this post because it uses an interesting dataset (a list of people who have Wikipedia pages by country) and he explains the process he did and the tools he used step by step - getting the data, choosing the model, optimizing it, fitting it to production constraints and deploying it. Real pleasure to read. http://nxn.se/post/127065307170/let-me-guess-where-youre-from Do you need a data scientist? I agree with many points in this post - sometimes data science and data engineering is mixed. I think that since "data scientist" become some kind of buzz word organizations might try to attract employees by this title. Regarding the point of having a problem to solve vs wanting to do something cool with data - yes, you have to have a concrete tasks for a person hired as a data scientist but you also have to have time for innovation and vision based on data and not all the time existing employees have those capabilities and knowledge. http://yanirseroussi.com/2015/08/24/you-dont-need-a-data-scientist-yet/ AWS CLI Jungle - this tool is kind of a wrapper to AWS CLI which wants to make it more intuitive. The lack of support in wild cards disturbs me as a user and I am glad they solve it. It currently implements functionality for ec2 and elb, I hope they will also implement s3 functionality soon. https://github.com/achiku/jungle

Redmonk programming language ranking

Mon, 24 Aug 2015 11:39:22 GMT

I usually don't find those ranking very interesting but Redmonk ranking which compares between programming languages' popularity on GitHub (by projects) and popularity on Stack Overflow (by questions) was interesting for me. What can we see in the chart below - very popular language (Java, Python, JavaScript, PHP) are very close to the y=x line - popularity on GitHub is almost the same as their popularity in Stack Overflow. Other languages which are very close to this line are - R, perl, Scala, Shell, Objective-C. Languages that are below the line are languages that are more popular in GitHub the in Stack Overflow.Very extreme example in this category is VimL (vim script). Why is that? one possible reason for that is that those languages are very well documented or that those languages have very limited use cases so the question scope is restricted. Another possible reason is the belief that there is no community, therefore people don't ask questions and don't get answers or ask in under another tag. For example VimL questions are sometimes asked with Vim tag and without VimL tag or a vimscript question. See this search. The languages above the line are languages which are more popular in Stack Overflow than in Github. Why is that? the opposite reasons - languages which are not documented very well or add new features frequently and therefore users have many questions about it. Another possibility is very common languages that are taught in the university and therefore many students which are not very experience ask a lot about them. A very extreme example for that is SQL. As well, XML is not a programming language and mostly used for configuration so it is quite clear why there are no projects in GitHub that are strictly XML. I think that both XQuery and DOT are above the line from the same reason. It was therefore surprising for me to see CSS very close to the line (btw could find HTML in the chart) but it possibly depends on the methodology of counting GitHub projects. Some of the languages are not really programming languages but rather a technology, e.g. Arduino, DOT, xml. It would be also interesting to see such a comparison to general technologies which are not programming languages - MongoDB, Elasticsearch, hadoop, etc. And to see it overtime - maybe documentation gets better, technology reaches to a stable state, etc.

Is Python random really random?

Thu, 20 Aug 2015 12:21:57 GMT

The question I started with, as the title states, was quite simple - "is python random really random"? Let's divide the question to two - python random and really random. Python random - a module from python standard library which implements pseudo-random number generators for various distributions. This module allows one to generate random integers in a given range, shuffle a sequence, take a sample from the sequence and so on. This post is a sanity check of whether I can trust python random or not. Really random - this is matter of statistics. If we have a coin and we tossed it 10 times and got 4 times head and 6 times tails what is our confidence that this coin is really fair? And if we got 8 times head and 2 times tails? or if we tossed it 100 times and got 40 times head and 60 times tails? So of course the bigger the sample is the higher the confidence we have whether the coin \ dice \ generator is random or not. This test actually asks if a random sequence python random produces has a mean like a "real" random distribution. A repeating sequence of 0, 1, 0, 1, ... will pass this test but it is clearly not random. [code language="python"] from random import randint from collections import Counter from math import sqrt, exp sizes = [10, 100, 1000, 10000, 100000] def normpdf(x, mean, sd): var = float(sd)**2 pi = 3.1415926 denom = (2*pi*var)**.5 num = exp(-(float(x)-float(mean))**2/(2*var)) return num/denom COIN_SIDES = 2 for size in sizes: sample = Counter([randint(1, COIN_SIDES) for i in xrange(0, size)]) expected_mean = size / COIN_SIDES expected_variance = size * \ (1.0 / COIN_SIDES) * \ (1 - 1.0/COIN_SIDES) expected_stdev = sqrt(expected_variance) p = 100 * normpdf(max(sample.values()), expected_mean, expected_stdev) q = 100 * normpdf(min(sample.values()), expected_mean, expected_stdev) print "size: %s, p(observed_mean > %s) = %s%%"\ %("{0:.3f}".format(1.0/COIN_SIDES), size, "{0:.3f}".format(p)) print "size: %s, p(observed_mean < %s) = %s%%"\ %("{0:.3f}".format(1.0/COIN_SIDES), size, "{0:.3f}".format(q)) print "samples: %s"%sample [/code] And the output - for sample size 100000 the probability that the observed mean is greater \ less than 0.5 is 0.251%. For p-value of 5%, i.e. the probability that the probability is less or great than 0.5 sample of size 1000+- is enough. As said before, it is a small sanity check but it passed it well.

5 interesting things (20/08/2015)

Thu, 20 Aug 2015 11:18:20 GMT

Beautiful Lies - in the good case charts can be unclear due to too much data, no distinct colors, missing labels etc. in a worse case it can be miss leading. Few tips and example of such -

http://flowingdata.com/2015/08/11/real-chart-rules-to-follow

RDBMS under the hood - a big part of this post deals with complexity and data structures. This introduction provides the ground to the rest of the post about different processes in the RDBMS, their costs and different optimizations. While I learned a big part of the post content as an undergraduate I wish I would also learn the more interesting part about optimizations.

http://coding-geek.com/how-databases-work/

Scaling and sharding @ pinterest - another link to a post about RDBMS. RDBMS is sometimes considered less sexy comparing to NoSQL, bigdata fancy solutions. This post explains why pinterest chose MySQL as their data warehouse solution and how they scale it. Comparing to the mambo jambo that happen in many companies the fact that this solution works for the for 3 years now is amazing.

https://engineering.pinterest.com/blog/sharding-pinterest-how-we-scaled-our-mysql-fleet/

What python cannot do - I was actually very optimistic about this post but ended up a bit disappointed. Some of the points it points on are relevant and the other I find less relevant. By less relevant points I refer to other programming languages with the same disadvantages - java and issues with versions. Several python modules lack good support - really? everywhere there is an open source (and also where there are commercial packages) documentation and quality are never perfect. "Errors can be identified only on run time" - dynamic typing can be both a pro and a con of python, it is a feature of the language. "Slower than other compiled languages" - this is miss leading as python is not compiled and comparing it C++ and C is weird. http://www.allaboutweb.biz/what-is-it-that-python-cannot-do/

Visual tutorial of Kalman filters - very interesting and well explained tutorial about Kalman filters.

http://www.bzarg.com/p/how-a-kalman-filter-works-in-pictures

5 interesting things (17/08/2015)

Mon, 17 Aug 2015 20:14:46 GMT

Hello Dask - Dask is a few month old addition to the PyData toolbox. It aims to enable light-weighted parallel computation. The idea behind it is that it produces a graph, which describes a data flow and is lazily evaluated. In this post Jake VanderPlas presents his usage of Dask on OpenStreetMap data and refer to Rob Story talk from PyData conference which presents the several tools in the PyData eco-system and big data tools (I usually don't have patience to see technical videos but this one is great). https://jakevdp.github.io/blog/2015/08/14/out-of-core-dataframes-in-python/

Benchmarks for ML implementations - Comparing between several open source implementations of machine learning algorithms for binary classification task. The data-set which was used for this task was the airlines data-set and the task was to predict whether a flight was delayed in more than 15 minutes. Results are interesting in several levels - the influence of bigger training set, scalebility of tools - Python fails on big training data, empirical comparison between different algorithms.

https://github.com/szilard/benchm-ml Folium - I have recently created some visualization based on geo-data and looked for the perfect (preferably python) tool for it. Matplotlib have basemap and I could also use some porting to d3.js or vincent but for all of them maps \ geo-data is just another type of data like bar chart and pie chart. Folium which is a wrapper of Leaflet.js focuses on interactive maps and does it well. https://github.com/python-visualization/folium Graph Databases for Beginners: Data Modeling Pitfalls to Avoid - Everything in this post is true and the pitfalls described there are indeed pitfalls when modeling data but I don't think they are unique pitfalls to graph database, they are also relevant to relational models.

http://neo4j.com/blog/data-modeling-pitfalls/

Data driven city - I'm fascinated by how public data can help improve the life of all of us. It can be governmental data about budgets to expose irregularities, data about air pollution, education, etc. This data is exists in most governments, government offices and municipalities and need to be formatted and clean. A work to put and maintain but it may be profitable. This field might not seem attractive to many entrepreneurs because it is not strictly business but there are many cities around the world fighting the same difficulties. I believe that the next big jump there is yet to be made and this post is one example of it.

http://blog.dominodatalab.com/optimizing-chicagos-services-with-the-power-of-analytics/

5 interesting things (10/08/2015)

Mon, 10 Aug 2015 15:17:41 GMT

rrrepo - at first glance it looked me like a checklist site and then I understood it is a repository of links. I tried to register, explore and create a site and I think they have many usability issues, some issue for example -

After registering I expected to get some mail confirming my registration and to automatically log into the site. It was unclear how to log into the site.
Wanted to create a repo in the name which already exists, wanted to navigate to it but I was redircted to somewhere else.
Exploring new repositories is not very clear.
Privacy settings.
When logged in, user is redirected to the news feed and cannot visit the home page or log out.

I have a bit of dejavu to urli.st which was closed a while ago but with a nicer UI. It is currently on very very early stage and many improvements to be done.

https://rrrepo.co/

Data Science blog list - I was like to explore new ideas and thoughts of others so this is a good starting point. Regarding the previous link (rrrepo) it can be a nice feature to import git repositories like this which is mainly links to a nice dashboard like rrrepo creates.

https://github.com/rushter/data-science-blogs

Composting music with recurrent neural networks - while traditional online music services recommend users existing music maybe we can just generate users new music?

http://www.hexahedria.com/2015/08/03/composing-music-with-recurrent-neural-networks/

Trust me I'm a data scientist? - Data scientist often explain their results to colleagues, users, clients which are not familiar with the full terminology of data science but want to make sure they can relay on the presented results. This post can make them trust you

https://lab.getbase.com/is-regression-trustworthy-or-how-to-use-metrics-to-trust-the-prediction-of-regression/

Streaming 101 - a 2 post series by O'reilly about streaming systems and data. This is only the first post in the series and it is packed with streaming related terminology, ideas and processing patterns. Good jump to the water for newbies.

http://radar.oreilly.com/2015/08/the-world-beyond-batch-streaming-101.html

5 interesting things (8/8/2015)

Sat, 08 Aug 2015 10:08:00 GMT

Nba shots charts - Well, I'm in love :) I like visualizations, I like sports and this post gives both the motivation and the how -

http://savvastjortjoglou.com/nba-shot-sharts.html

NBA players KNN - going on with the sports theme Vik Paruchuri tried to predict the points NBA player will contribute based on his nearest neighbors. I find this analysis a bit missing - first his results are not there, was this process successful or not? He looks for the most similar players to Lebron James but does not show who they are, for me that's interesting to know, some gut feeling about the result. He uses only numeric columns which is ok, but there are also categorical variables (e.g position), why does he use them as well \ how can he use them in the future? And last but not least some visualization - for each player show me the KNN, some histogram regarding the prediction vs true values. Something.. I feel he wrote only half a post..

https://www.dataquest.io/blog/k-nearest-neighbors

Rpython and pandasshell - as I see it both tools make a step toward the other. That's an interesting questions whether they both survive. As a software developer I don't usually see that need to switch to R. https://github.com/robdmc/pandashells http://kirbyfan64.github.io/posts/the-magic-of-rpython.html http://rpython.readthedocs.org/en/latest/

Wining as a service - twitter vs twitter, using twitter power (+ coding + creativity :) to win only twitter lotteries and competitions.

http://www.hscott.net/twitter-contest-winning-as-a-service/

Engineers to managers - this post aroused too much thoughts to TL;DR it in only few sentences. I'm not sure it the timeline is correct or if I'll adopt it myself but the ideas and changes about becoming a manager are inspiring and real food for thought for me.

http://firstround.com/review/this-90-day-plan-turns-engineers-into-remarkable-managers/

Music Information Retrieval from a Multicultural Perspective

Fri, 07 Aug 2015 09:44:08 GMT

Yesterday I attended a "Music Information Retrieval from a Multicultural Perspective" meetup in SoundCloud offices. It was one of the most thought provoking meetups I have attended in a long time. Prof Xavier Serra, who gave the talk has a lot of experience with Music Information Retrieval and with community projects along with industrial projects - dunya, acousticbrainz, compmusic and more.

For me music information retrieval is completely new domain. I had several thoughts in the past about it such as - when you upload a clip to youtube how do the make sure you don't violate any copyright? how do you do it fast and scaleable? How do you recommend someone a new music? Can you just design it like any other recommendation system? How to process the signal to differentiate and recognize the different instruments and so on.

As a data scientist and software engineer it is clear that when choosing features, which features to extract, we are skewed. We have a gut feeling and we usually follow that. Even when choosing which algorithms to use or which tool to use we usually turn either to something we already succeeded with and have experience or to a new algorithm \ technology we want to learn. We are sometimes blind to those biases.

Big parts of software development and data science is done on the western world, i.e US, Europe, etc. there are cultural biases to the engineers and the way they think. Our gut feeling is biased. In this case our gut feeling is biased toward how we in the western world understand what music is and how it is built - scores, notes, intonation, etc. Giving the examples in the lecture and stories behind them, it is clear that the way music is interpreted and experienced different in different cultures.

It is quite shocking to think about science that way - although we think we are quite rational and following some scientific \ research \ independent process we are biased to begin with. The implications for me are immediate - how recommendation system work (not only music). Simply think of movies, maybe the categorization to genres is external to other cultures? Maybe they care about different features in products?

I'll try to bare it in mind next time I design a recommendation system.

Data Driven - creating a data culture

Mon, 03 Aug 2015 09:48:26 GMT

I have just read "Data Driven - creating a data culture" by Hilary Mason and DJ Patil. While computer science is considered a new science or a new discipline data science is even a newer one. Well is it? Is it a discipline? What is the goals of data scientist and data science team within different organizations? Where should they place? What are their responsibilities and how is success measured? This remained me the course "Scientific Thinking" I took as an undergrad and specifically Weber's "Wissenschaft als Beruf" and Kuhn's "The Structure of Scientific Revolutions". For me it seems that data science is in the phase of Kuhn's pre-paradigm and Mason and Patil tries to move it in to the phase of normal science. They argue about the place and limitation of data science, what are the characteristics of data scientists. Maybe most importantly they talk about how to measure success and how to set goals to data science process. So what about Weber? In this lecture, Weber talks about the advantages of choosing an academic career versus an industry career. He also compare between the university systems in Europe and in the US. Almost 100 years has passed since then and the world slightly changed, including the role of work in our life. I think that for many people working in the industry as data scientist has both the advantages of being a scientist (doing some research) and working in the industry (salary, technology)... But I'm not sure if it is a really a "science". For me personally this title or definition is not important - I love what I do and it creates a measurable value to the organizations I work for. I see many of the processes suggested in this book as trying to bridge between and academic life and the industry life. Over all, this hand book was ok, but I expected a bit more.

5 interesting things (30/07/2015)

Thu, 30 Jul 2015 10:55:36 GMT

Python neural network in 11 lines - the name is a bit misleading but nice DIY initiative.

http://iamtrask.github.io/2015/07/12/basic-python-network/

Python or R - The bottom line for me is that both are tools and you should adjust the tool you use to the task. I personally feel more comfortable on Python and really glad that ideas, prototypes and thoughts I had can move to production using almost the same code and does not require additional work.

http://www.infoworld.com/article/2951779/application-development/in-data-science-the-r-language-is-swallowing-python.html

Python, Ruby and Golang web framework comparison - I think that for most of the people and projects this comparison is not relevant as either they are experts \ using a specific language or the language is limited by the project environment. But for the cases it is relevant - switching between language, learning a language and POCing it is nice to have a comparison with frameworks you know, feel comfort with.

https://realpython.com/blog/python/python-ruby-and-golang-a-web-Service-application-comparison/

Pandas and Apache Spark data frames differences - Both Pandas and Apache Spark are very common tools for data scientist using Python. It is therefore very important to know the differences between the data frames in both tools as one may assume it might function the same.

https://medium.com/@chris_bour/6-differences-between-pandas-and-spark-dataframes-1380cec394d2

Anti patterns - at first glance I thought it was a very weird idea but then going over several examples and it was actually interesting and enlightening. Some things I do automatically and haven't thought of them for a long time, some matters of style, etc. Interesting although I was skeptic in the beginning :)

http://docs.quantifiedcode.com/python-code-patterns/index.html

5 interesting things (23/07/2015)

Thu, 23 Jul 2015 15:15:34 GMT

Diversity in tech? This post describe a black woman’s experience in the Stanford Computer Science Major. As part of TOA events I went to Zalando's diversity in tech panel last week. So I got to think about this subject for a while both from my situation - foreign women in tech, not speaking the local language and from a wider point of view. Having women in Universities (and any other population) is a necessary condition (but not sufficient) to having diversity in tech.

http://theodysseyonline.com/stanford/black-women-in-tech/128310

Invitation to Scala - this post tries to make Scala less intimidating for Scala beginners. May the force be with him (and with me)

https://www.voxxed.com/blog/2015/07/scala-doesnt-complicated/

Pyxley - Python powered dashboards. I'm always excited about visualization tools. It is built with Pandas data frame in mind and therefore should be relatively intuitive for data scientist which uses python.

http://multithreaded.stitchfix.com/blog/2015/07/16/pyxley/

Cloudera Ibis - Cloudera reveals Ibis project which is aimed to give python end-to-end pipelines specially for data scientist with in the well known PyLab eco-system (pandas, scikit-learn, scipy, etc).

http://blog.cloudera.com/blog/2015/07/ibis-on-impala-python-at-scale-for-data-science/

http://www.ibis-project.org/

Clustering check-ins with Spark and Cassandra - the title is self explanatory... Loading check-ins data data to Cassandra, analyzing it with Spark and visualizing it with zeppelin. All in all a reasonable data product pipeline put together beautifully

http://www.natalinobusa.com/2015/07/clustering-check-ins-with-spark-and.html

Database debate

Thu, 16 Jul 2015 22:07:26 GMT

As part of the Berlin Tech Open Air events I want to the "Database debate" in DC Media networks. The two sides of the debate were Simon Willnauer, lead engineer of ElasticSearch and co-founder of Elasticand Carter Page, technical lead for Google BigTable.

The talk was hosted by DC Media networks employee and was navigated really well with nice questions and interactions between Willnauer and Page. However it was not a "Database debate" at least for two reasons. ElasticSearch currently does not brand itself as a database and as admitted by Willnauer is not mature enough. Therefore it was not a debate - no pros and cons, no one against the other. But rather two solutions for different problems which both somehow relate to the buzzword "big data".

Some expected questions were asked - use-cases, road map and future features, bug fixes, comparison to other solutions. But also less expected questions - what would you do different if rebuilding the product, pitfalls of beginners and some more technical deep dive questions asked by the audience.

One of the nice questions asked by the host was - "What is the weirdest usage you have seen to your product". Willnauer answered - "playing chess with it using near neighbor to compute the next step". Following Call me maybe project I though of "Call me checkmate" project - playing chess using different databases. Overall, very nice and chill atmosphere. Although I'm not sure why it is called open air..

5 interesting things (12/07/2015)

Sun, 12 Jul 2015 14:29:12 GMT

Code management tools by AWS - in the last RE:INVENT event (October 2014) Amazon said that this year they are going to focus on new tools for code management and deployment. Now they reveal those tools -

https://aws.amazon.com/blogs/aws/code-management-and-deployment/

Mail received from the closest Oak tree - I find this blig post and the whole process charming. The city of Melbourne created a technology interface to get the citizens more involved in the city life and exciting things happened. I find those interactions between the everyday life and the public sphere as one of the most fascinating challenges of the coming years - making the public sphere more accessible, smart and open.

http://www.citylab.com/tech/2015/07/when-you-give-a-tree-an-email-address/398219/

Toyplot - another python plotting library. Seems to work natively with Numpy. Still quite young - number of different possible charts is limited but I'm it will become more mature in the near future. Beside nice, interactive charts which I'm able to configure to my needs (axes, legend, colors, scale, exporting \ embedding visualizations etc) what I look for in a good plotting package is answer all the different type of charts I need. I don't want to start juggling between several packages each for a different type. At least on this area there is always a place to grow - heat maps, geo-spatial maps, 3d, etc.

http://toyplot.readthedocs.org/

Python design patterns - I'm Tom and I'm lazy, I admit it. I think I said it before but IMHO a good software developer is a lazy one. One who automates what she can, uses existing tools and packages when available and reuses her own code. This is the main task of design patterns - solve common problems and provide best practices. And also create common language so different developers and stakeholders can communicate. This github repository collects design patterns implementations in Python.

https://github.com/faif/python-patterns

Mining twitter data with Python - a seven posts series by Marco Bonzanini. Goes through the entire process starting with getting twitter access token and ends with data visualization using d3.js and sentiment analysis.

http://marcobonzanini.com/2015/03/02/mining-twitter-data-with-python-part-1/

5 interesting things (10/07/2015)

Fri, 10 Jul 2015 10:17:31 GMT

Three Useful Python Libraries for Startups - tl;dr this post suggest Whitenoise, Phonenumbers and Pdfki as 3 important packages for startups and requests and Python-dateutil as runner-ups. IMHO, those are very strange choices. I agree with choosing requests as a package that simplify http requests and which is important to infrastructure. I expected the other packages as well to relate to infrastructure. Possibly libraries I would think of - django and boto. Possibly also numpy and \ or pandas for very common statistic and analysis use cases. http://blog.instavest.com/three-useful-python-libraries-for-startups Trending @ Instagram - I used to work on a very similar problem to this one and facing almost the same challenges - ranking and scoring, grouping. It is always interesting to see how different people approach the same problem which I know intimately. http://instagram-engineering.tumblr.com/post/122961624217/trending-at-instagram Git from the inside out - version control is very important tool in the everyday life so it is nice to look into one possible implementation of it. https://codewords.recurse.com/issues/two/git-from-the-inside-out Document clustering with Python - simple, clear howto guide which both explain the theory lightly, examine several clustering algorithms and sums up with visualizations. http://nbviewer.ipython.org/github/brandomr/document_cluster/blob/master/cluster_analysis_web.ipynb Deploying python packages @ Nylas - I love such posts which explain the real life problem they faced, suggest several solution \ possible alternative and their pros and cons and show what and why they eventually choose. Specifically as I believe every python developer run onto those problems at least once (a day :)) https://nylas.com/blog/packaging-deploying-python

5 interesting things (26/06/2015)

Fri, 26 Jun 2015 16:10:56 GMT

Spark Summit 2015 Highlights in Tweets - Spark summit was held in San Francisco this week (15-17 June). Euro-Spark summit will be held on 27-29 October call for papers is open. This presentation is a brief of the highlights of different talks. I believe that (if not already) most of the talks and presentations will be available online soon.

http://www.slideshare.net/maasg/spark-summit-2015-highlights

How to implement neural network - somehow, neural networks are not fully supported by scikit-learn (probably will happen in the future) and there are several alternative such as PyBrain (not sure if it is maintained), PyLearn2, theano and so on.This tutorial is Neural Network DIY. It is both a technical-python guide and a step-by-step neural network reminder.http://peterroelants.github.io/posts/neural_network_implementation_part01/ Designing large scale python application - Talk by M.A Lumberg, Python core developer, in PyWaw summit. He also talked as a keynote about - "Past, present and future of Python", I intend to hear that one when I have sometime. http://www.egenix.com/library/presentations/PyWaw-Summit-2015-Designing-Large-Scale-Applications-in-Python/ The artpiece of command line - I'm a geek, I admit it and I like working from the command line (we love you Tom). Both for small data science things (sum, average, uniq, etc.) and for checking up what is going on in my system (atop, ps, etc.). I fell I should keep this repository not more than a click a way. https://github.com/jlevy/the-art-of-command-line Mattermost - mattermost is an open source alternative for Slack. In one of the organizations I worked introducing Slack changed the entire communication in the organization. People were sharing more of their ideas, thoughts, results and got involved into additional projects. Still some features made me feel inconvenient like preferring Slack over documentation, using external servers and hard to search \ backup. I therefore find it very exciting to have an open source alternative that will improve both products. http://www.mattermost.org/

5 interesting things (19/06/2015)

Fri, 19 Jun 2015 17:36:16 GMT

Interviewing as an art - Finding the smart, talented people with the right vibe for your company is always hard. Being on the other side is also hard. Not only in the sense of being good enough but also "interviewing" the work place - is the job interesting \ challenging? are the colleagues nice, talented, will you like to work with them? etc. There are many posts and theories about what you should ask in order to find the right candidate and less posts and discussions about how to approach the candidate, make him or her comparable and being able to get the maximum out of it during a stressing interview. I find this post specially sensitive and inviting.

http://www.zdfs.com/code/2015/on-interviewing-software-engineers

Airbnb airflow - a framework by AirBnb which help create, schedule and monitor data pipelines. https://github.com/airbnb/airflow

Seaborn cheat-sheet - the statistical data visualization package completing \ improving the PyData eco-system. A simple tutorial that can also be used as a cheat-sheet. https://beta.oreilly.com/learning/data-visualization-with-seaborn

Data Science Ipython notebooks - a collection of links of Ipython notebooks which relates to data science. Include notebooks from Kaggle, AWS, PyCon, etc. Can also add a the link to some seaborn notebook.. https://github.com/donnemartin/data-science-ipython-notebooks

Pitfalls of A\B testing - tl;dr - statistics is tricky and you should remember the underlining assumptions. Also a good chance to remind A/A testing as a sanity check. http://blog.dato.com/how-to-evaluate-machine-learning-models-the-pitfalls-of-ab-testing

5 interesting things (04/06/2015)

Thu, 04 Jun 2015 15:33:07 GMT

Pitfalls when mining Wikipedia - this research was done on a bit old data (2013) but it show quite easy way to clean wikipedia dumps. Although they suggest a way to repeat the process they did, I suspect that the structure of some of the dumps was changed a bit. An interesting question for me in this scope is the relation between content \ views of same entities cross different languages. This can be calculated by joining page count results with wikidata dumps once they are complete. Crunching wikidata dumps a bit, at the moment it does not feel mature enough and does not include the data about all the articles in wikipedia. https://blog.lateral.io/2015/06/the-unknown-perils-of-mining-wikipedia/ Weather prediction with Amazon Machine Learning - nice, simple way to start playing with Amazon machine learning - http://arnesund.com/2015/05/31/using-amazon-machine-learning-to-predict-the-weather/ How to evaluate machine learning models - a series of 5 posts by dato - http://blog.dato.com/how-to-evaluate-machine-learning-models-part-1-orientation Mean shift clustering - clustering approach with few advantages over common methods. E.g - don't have to define number of clusters in advance - but you have to tune the bandwidth. On the other hand it slower than other algorithms.

http://spin.atomicobject.com/2015/05/26/mean-shift-clustering

The future of spark - post following Strata + Hadoop conference in London. https://www.linkedin.com/pulse/future-apache-spark-rodrigo-rivera

5 interesting things (31/05/2015)

Sun, 31 May 2015 16:18:14 GMT

Vis.js demo - (Yet another) Javascript visualization library. I love the relative graph approach, I find it easier for Python developers that d3.js (which I also like) -

http://chris.neoxygen.io/articles/easy-graph-visualization-with-vis-dot-js.html

Predition.io - a machine learning server for developers and data scientists. Built on Apache Spark, HBase and Spray. Haven't used it but it seems very interesting -

https://prediction.io/

Django performance - 4 litlle tricks - I'm familiar and experienced with Django but not deep dive. Possibly those are very basic tricks but for me they were good to keep in mind

http://www.revsys.com/blog/2015/may/06/django-performance-simple-things/

Deep dive into elasticsearch storage - as I currently work a lot with elasticsearch (should say elastic..) I find this post very interesting

https://www.found.no/foundation/dive-into-elasticsearch-storage/

Amazon machine learning - the new buzz in the neighborhood

https://aws.amazon.com/machine-learning/

6 interesting things (20/2/2015)

Fri, 20 Feb 2015 12:00:39 GMT

Making an exception but I really came across some interesting things -

A\A Testing - well known idea in machine learning (train, cross-validation, test) and in other research best practices now used as a take off on the buzz word term - "A\B testing" but yet well explained and can be eye opening on the right moment -

http://jvns.ca/blog/2015/02/06/a-a-testing/

Intro to into - Easier conversion between somehow complex data types in python

http://matthewrocklin.com/blog/work/2015/02/03/Into/

Reading this I was also exposed

http://blaze.pydata.org/docs/dev/index.html

Getting started with Spark in Python - very very clear tutorial about all the required steps to get started. I cannot wait to find a good enough excuse to work with spark.

https://districtdatalabs.silvrback.com/getting-started-with-spark-in-python

Fuzzy - Fast Python phonetic algorithms. Nothing new or too fancy, just came across it this week and found it useful.

https://pypi.python.org/pypi/Fuzzy

Typo Distance - finds typo distance between two strings. It uses qwerty layout but you can configure different layout pretty easily. The algorithm is quite heavy and time consuming, there is some room for improvement (although it is not actively maintained). For example - adding a max parameter which stops the computation once the typo distance is higher than the allowed distance.

https://github.com/wsong/Typo-Distance/blob/master/typodistance.py

Topy - Python script to fix typos in text, using rule-sets developed by theRegExTypoFix project from Wikipedia. The basic rule set is an English rule set but other rule sets are also available. Trying it, I'm positive about it but it is not baked \ mature enough and I would like it to be more easy to use in code than as a command line tool. https://github.com/intgr/topy

5 interesting things (03/02/2015)

Tue, 03 Feb 2015 18:37:48 GMT

spaCy - yet another python library for text processing? maybe, haven't tried it but seems like another tool taking part in this growing world. I'm looking for a package that will do a good job on short texts such as tweets and so on.

http://honnibal.github.io/spaCy/

Moto - not what you thought.. not a short from automotive but mocking +boto. "A library that allows your python tests to easily mock out the boto library". Looks very neat but still has some way to go.

https://github.com/spulec/moto

Nothing like CLI - examples that for certain use cases command line tools can be much faster than hadoop cluster.

http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html

Self hell - elegance in code although I'm not sure how efficient it is.

http://kmkeen.com/self-hell/

NLP made visual with Neo4j - the concept is cool - creating an interesting visualization not with the trivial \ designated tool for it. I doubt how much it scales.

http://jexp.de/blog/2015/01/natural-language-analytics-made-simple-and-visual-with-neo4j/

5 interesting things (04/01/2015)

Sun, 04 Jan 2015 15:21:31 GMT

Is Wikipedia a microcsmos of our world? News of 2014 as they are reflected in Wikipedia.

http://www.brianckeegan.com/2014/12/the-news-on-wikipedia-in-2014/

Json as python modules - Making life simpler. Making it possible to do -"import x" for x.json.

https://github.com/kragniz/json-sempai

Python the fast way - living on the edge with Python :) Some hints on how to make your code run faster. Another possibility - checking different interpreters. It was nice to see the assembly commands, I was expecting something more advanced.

http://pythonfasterway.uni.me/

Data Science Ontology - just having fun with d3.js.

http://www.datascienceontology.com/

Nips Experiment - there was relatively a lot of chatter regarding the experiment done by NIPS committee. Non the less, it creates a very uncomfortable feeling regarding committees.

http://mrtz.org/blog/the-nips-experiment/

5 interesting things (14/12/2014)

Sun, 14 Dec 2014 16:56:24 GMT

Twitter analytics with Spark - I really enjoyed this post from several reasons. First because the need and the way that lead to the solution is clear and well explained. Second, it uses a novel approach, mining social network for non trivial uses. Moreover, querying the app resulted a very nice results which is not trivial. I missed some reasoning \ explanation in the app. I.e., better understanding the connections between two organizations. I also wonder about the velocity the relations change. The technical part was interesting as well although I haven't got to work with Spark so far.

http://eugenezhulenev.com/blog/2014/11/20/twitter-analytics-with-spark/

Teaching creative - I really love to teach and I also try to adjust the methods, content, example to the audience and to be as creative as I can and to be passionate about the content. I believe that this way the audience will remember what I talked about. Also to be hands on help to remember and understand. This post which demonstrates creativity on so many levels is amazing and is very inspiring.

http://blog.mattwaite.com/post/103144761014/a-classroom-experiment-in-twitter-bots-and

Databases compression - Summarizes the high level and things to know about each database very well. A good starting point when evaluating several solutions.

http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis

A bonus track - http://blog.nahurst.com/visual-guide-to-nosql-systems

Experiments at AirBnB - Although I usually prefer to read about private people that did something cool with data I like AirBnB's data blog as there is always something interesting to read about. This time it is about experiments, the very trendy "AB testing". It raises

http://nerds.airbnb.com/experiments-at-airbnb/

DevOps bookmarks - devops tools and frameworks. An aggregator, reminds assume * but very tempting implementation.

http://www.devopsbookmarks.com/

Common Crawl meetup

Tue, 18 Nov 2014 19:40:52 GMT

Yesterday I attended big data beers meetup. The meetup included 2 talks by Common Crawl employees - Lisa Green and Stephen Merity. Both talks was great and the connection between them was empowering.

The meetup was sponsored by Data Artisans which are working on Apache Flink. Too bad I don't have time to go to their meet up today.

What is Common Crawl?

Common Crawl is a NGO that makes web data accessible to everyone with little or not cost. They crawl the web, release a monthly build which is stored in AWS S3 under public data sets. They respect no robots and no follow flags and basically try to be good citizens in the internet cosmos. As Lisa Green said in her talk - they believe that the web is "a digital copy of our world" and the greatest data set and their mission is to make it available to everyone.

Technicalities

Monthly build (currently prefer bigger monthly builds over more frequent builds)
Latest build was 220TB with crawl data of 2.98 billion web pages.
Data include 3 type of files -
- WARC files of the raw crawl data
- WAT files which include the metadata for the data stored in the WARC (about 1/3 of the raw crawl data)
- WET files which hold the plaintext from the data stored in the WARC (about 15% of the raw crawl data).
Delay between publication of page and crawl \ public time is approx month-month and a half.
No incremental dumps are planned at the moment.
The data is currently skewed to dot com domains. They plan to improve it changes will hopefully be seen on January dump.
They crawl using Apache Nutch - Nutch is an open source web crawler and cooperate with Blekko in order to avoid spam.

Common Crawl encourages researchers, universities and commercial companies to use their data. If you ask politely they will even grant you some Amazon credit.

The Talks

Lisa Green talked about the general idea of open data (governments data, commercial data, web data) and gave some examples for using open data in general. The example I liked the most - using Orange cell phone data to identify Ebola spreading patterns, see here. This was a very inspiring introduction to Stephen Merity more technical talk.

Stephen Merity spoke about the more technical parts and gave an amazing examples on how to do both fast and cheap computations (spot instances rock). He showed interesting data about computing PageRank on the entire Common Crawl data, some NLP stuff and other interesting insights about their data.

Another relevant talk in the area is Jordan Mendelson talk from Berlin BuzzWords - "Big Data for Cheapskates" (if your are on a hurry start from the 18th minute).

Slides are available in -

Stephen Merity slides

What can you do with Common Crawl

Treating it as a data set there is a lot to explore -

1. Train it for language detection - train it for language detection for specific domains.

2. Named Entity Recognition.

3. Investigate the relations between different domains, web structre - identify competitors, page rank, etc.

4. Investigate the relations between different technologies - which js libraries appear together, changes of technology usage over time.

5 interesting things (06/11/2014)

Thu, 06 Nov 2014 20:42:21 GMT

Geeks pleasure - fulling math. Creating two images (and actually every type of document) with the same md5 hash.

http://natmchugh.blogspot.de/2014/10/how-i-created-two-images-with-same-md5.html

Django vs Flask vs Pyramid - Python has a great open source community which is growing rapidly. One of the advantages is having several solution to the same or to near by problems. This post compares between 3 well known Python web frameworks - Django, Flask and Pyramid.

From my point of view working with Django and Flask. Flask is like riding a motorcycle while Django is Tank. Django is more tightly coupled with SQL backedend and the relevant dependencies and plugins while Flask allow quick, light functionality.

https://www.airpair.com/python/posts/django-flask-pyramid

The Science of Crawl - half related to a project I currently do. Those two posts concern to problems that everyone how indexed and \ or crawled data faced with.

http://blog.urx.com/urx-blog/2014/9/4/the-science-of-crawl-part-1-deduplication-of-web-content

http://blog.urx.com/urx-blog/2014/10/23/the-science-of-crawl-part-2-content-freshness

The invisible wall - 25 years later and the invisible wall still separates east from west in Germany. Beside living in Berlin visualizations convey the point very well.

http://www.washingtonpost.com/blogs/worldviews/wp/2014/10/31/the-berlin-wall-fell-25-years-ago-but-germany-is-still-divided/ Toolz - we all know the term design patterns. Toolz provides implementation pattern. Everyday utilities every developer needs - iterators, dictionaries, etc. Two links - blog post with common use cases and Toolz documentation. http://matthewrocklin.com/blog/work/2014/07/04/Streaming-Analytics/ http://toolz.readthedocs.org/en/latest/

Running my first EMR - lessons learned

Wed, 15 Oct 2014 17:34:46 GMT

Today I was trying to run my first EMR, here are few lessons I learned during this day. I have previously run hadoop streaming mapreduce so I was familiar with the mapreduce state of mind. However, I was not familiar with the EMR environment. I used boto - Amazon official python interface.

1. AMI version - default AMI version is 1.0.0 - first release. This means the following specifications -

Operating system: Debian 5.0 (Lenny) Applications: Hadoop 0.20 and 0.18 (default); Hive 0.5, 0.7 (default), 0.7.1; Pig 0.3 (on Hadoop 0.18), 0.6 (on Hadoop 0.20) Languages: Perl 5.10.0, PHP 5.2.6, Python 2.5.2, R 2.7.1, Ruby 1.8.7 File system: ext3 for root and ephemeral Kernel: Red Hat

http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/ami-versions-supported.html

For me Python 2.5.2 means -

Does not include json - new in version 2.6.

collections is new in python 2.4, but not all the models were added in this version -

namedtuple()	factory function for creating tuple subclasses with named fields	New in version 2.6.
deque	list-like container with fast appends and pops on either end	New in version 2.4.
Counter	dict subclass for counting hashable objects	New in version 2.7.
OrderedDict	dict subclass that remembers the order entries were added	New in version 2.7.
defaultdict	dict subclass that calls a factory function to supply missing values	New in version 2.5.

dict comprehensions is also kind of late addition (python 2.7)

Therefore specifying the ami_version version can be critical. Version 2.2.0 worked fine for me.

2. Must process all the input!

Naturally we will want to process all the input. However, for testing I went over only the n-first lines and then added a break to make things run faster. I was not consuming all the lines and therefore got an error. More about it here -

http://stackoverflow.com/questions/9881269/broken-pipe-error-causes-streaming-elastic-mapreduce-job-on-aws-to-fail

3. Output folder must not exists. This is the same as in hadoop streaming map reduce, for me the way to avoid it was to add a timestamp - [sourcecode language="python" wraplines="false" collapse="false"] output="s3n://<my-bucket>/output/"+str(int(time.time())) [/sourcecode]

4. Why my process failed - one option which produces are relatively understandable explanation is - conn.describe_jobflow(jobid).laststatechangereason

5. cache_files - enables you to import files you need for the map reduce process. Super important to "specify a fragment", i.e. specify the local file name

[sourcecode language="python" wraplines="false" collapse="false"] cache_files=['s3n://<file-location>/<file-name>#<local-file-name>'] [/sourcecode]

Otherwise you will obtain the following error -

"Streaming cacheFile and cacheArchive must specify a fragment"

6. Status - the are 7 different status your flow may have - COMPLETED, FAILED, TERMINATED, RUNNING, SHUTTING_DOWN, STARTING and WAITING. The right order of statuses if everything goes well is STARTING -> RUNNING ->SHUTTING_DOWN -> COMPLETED.

The SHUTTING_DOWN may take a while even for a very simple flow I measured about 1 minute of SHUTTING_DOWN process.

Resources I used -

http://boto.readthedocs.org/en/latest/emr_tut.html

http://atbrox.com/2010/10/01/programmatic-deployment-to-elastic-mapreduce-with-boto-and-bootstrap-action/

5 interesting things (04/09/2014)

Thu, 04 Sep 2014 10:26:25 GMT

C3.JS - I have previously wrote a post about the importance of visualization in the skill set of data scientist. C3.js is a JavaScript chart library based on d3.js which seems at least in a glimpse to be simple and intuitive. I would like to see a Python client for that but that for the future to come.

http://c3js.org/

nvd3 also do something like - charts based 3d.js and also have a Python client which I worked with a bit. Comparing the two c3.js seems a little bit more mature than nvd3, ignoring the lack of Python client but I'm sure that gap would be filled soon.

nvd3.org

Harvard Visualization course - I went through some of the slides and it was fascinating but what is even more exciting is the great collection of links about visualization examples, theory and tools. Great work.

http://www.cs171.org/#!index.md

textract - I needed high flexibility of input types in a project I do and of course I wanted to deal with as transparent as possible without looking myself to all the relevant packages or adjust my code to the API of each package. Fortunately somebody already did it - textarct. The package is not perfect and there are some "glitches" mostly concerning the parsers themselves (line splitting, non-ascii, etc) and not to the unified API textract provides. However, it is a very good start.

http://textract.readthedocs.org/en/latest/

Visualizing Garbage Collection Algorithms - both very cool visualization and good explanations. Design wise I think the visualization should be larger but the concept itself is very neat.

http://spin.atomicobject.com/2014/09/03/visualizing-garbage-collection-algorithms/

SmartCSV - making CSV reader more structured by defining a model and validating it while reading. Enables skipping rows (and soon skipping also columns). It is on going project and feature requests and issues are currently addressed quickly.

https://github.com/santiagobasulto/smartcsv

5 interesting things (26/08/2014)

Tue, 26 Aug 2014 08:58:33 GMT

Gooey - Command line to application! Very cool. Works by mapping ArgumentParser to GUI objects. Very cool and make it easier to make instant tools to play around the office. Future tasks include supporting in more themes, different arg parses (e.g docopt http://docopt.org/), etc. I believe there is much potential in this project.

https://github.com/chriskiehl/Gooey

Datamesh - stats command line tool. I like working for the command line, at least to begin with and to feel data before doing complex things, this tool is really what I need. However, the documentation is missing \ not clear enough \ open source style.

http://www.gnu.org/software/datamash/

In that sense, q, http://harelba.github.io/q/, is also very cool and answer the need to things and speak in the same framework in different tasks.

Getting started with Python internals - how to dive to the deep water. Very enriching post which also include links to other interesting posts.

http://akaptur.github.io/blog/2014/08/03/getting-started-with-python-internals/

python-ftfy - unicode strings can be very painful when it includes special characters as umlauts (ö), html tags (>) and everything users can think of :). The goal of this package is to at least partly ease this pain.

https://github.com/LuminosoInsight/python-ftfy

Fraud detection Lyft vs Uber - I think that the interesting part here is the data visualization as tool to understand the problem better.

http://linkurio.us/lyft-vs-uber-visualizing-fraud-patterns/

setdefault vs get vs defaultdict

Mon, 25 Aug 2014 17:55:49 GMT

You have a python dictionary, you want to get the value of specific key in the dictionary, so far so good, right?

And then a KeyError -

Traceback (most recent call last): File "<stdin>", line 1, in <module> KeyError: 1

Hmmm, well if this key does not exist in the dictionary I can use some default value like None, 10, empty string. What's my options of doing so?

I can think of 3 -

get method
setdefault method
defaultdict data structure get method

Let's investigate first - [sourcecode language="python" wraplines="false" collapse="false"] key, value = "key", "value" data = {} x = data.get(key,value) print x, data #value {} data= {} x = data.setdefault(key,value) print x, data #value {'key': 'value'} [/sourcecode] Well, we get almost the same result, x obtains the same value and in get data is not changed while in setdefault data changes. When does it become a problem? [sourcecode language="python" wraplines="false" collapse="false"] key, value = "key", "value" data = {} x = data.get(key,[])append(value) print x, data #None {} data= {} x = data.setdefault(key,[]).append(value) print x, data None {'key': ['value']} [/sourcecode] So, when we are dealing with mutable data types the difference is clearer and error prone. When to use each? mainly depends on the content of your dictionary and its' size. We can time the differences but it does not really matter as they produce different output and it was not significant for any direction anyhow. And for defaultdict - [sourcecode language="python" wraplines="false" collapse="false"] from collections import defaultdict data = defaultdict(list) print data[key] #[] data[key].append(value) print data[key] #['value'] [/sourcecode] setdefault sets the default value to a specific key we access to while defaultdict is the type of the data variable and set this default value to every key we access to. So, if we get roughly the same result I timed the processes for several dictionary sizes (left most column) and run each 1000 times (code below) -

dict size	default value	method	time
100	list	setdefault	0.0229508876801
defaultdict	0.0204179286957
set	setdefault	0.0209970474243
defaultdict	0.0194549560547
int	setdefault	0.0236239433289
defaultdict	0.0225579738617
string	setdefault	0.020693063736
defaultdict	0.0240340232849
10000	list	setdefault	2.09283614159
defaultdict	2.31266093254
set	setdefault	2.12825512886
defaultdict	3.43549799919
int	setdefault	2.04997992516
defaultdict	1.87312483788
""	setdefault	2.05423784256
defaultdict	1.93679213524
100000	list	setdefault	22.4799249172
defaultdict	29.7850298882
set	setdefault	23.5321040154
defaultdict	41.7523541451
int	setdefault	26.6693091393
defaultdict	23.1293339729
string	setdefault	26.4119689465
defaultdict	23.6694099903

Conclusions and summary -

Working with sets is almost always more expensive time-wise than working with lists
As the dictionary size grows simple types - string and int perform better with defaultdict then with setdefault while set and list perform worse.
Main conclusion - choosing between defaultdict and setdefault also mainly depends in the type of the default value.
In this test I tested a particular use case - accessing each key twice. Different use cases \ distributions such as assignment, accessing to the same key over and over again, etc. may have different properties.
There is no firm conclusion here just investigating some of interpreter capabilities.

Code - [sourcecode language="python" wraplines="false" collapse="false"] import timeit from collections import defaultdict from itertools import product def measure_setdefault(n, defaultvalue): data = {} for i in xrange(0,n): x = data.setdefault(i,defaultvalue) for i in xrange(0,n): x = data.setdefault(i,defaultvalue) def measure_defaultdict(n,defaultvalue): data = defaultdict(type(defaultvalue)) for i in xrange(0,n): x = data[i] for i in xrange(0,n): x = data[i] if __name__ == '__main__': import timeit number = 1000 dict_sizes = [100,10000, 100000] defaultvalues = [[], 0, "", set()] for dict_size, defaultvalue in product(dict_sizes, defaultvalues): print "dict_size: ", dict_size, " defaultvalue: ", type(defaultvalue) print "\tsetdefault:", timeit.timeit("measure_setdefault(dict_size, defaultvalue)", setup="from __main__ import measure_setdefault, dict_size, defaultvalue", number=number) print "\\tdefaultdict:", timeit.timeit("measure_defaultdict(dict_size, defaultvalue)", setup="from __main__ import measure_defaultdict, dict_size, defaultvalue", number=number) [/sourcecode]

EuroPython 2014 - Python under the hood

Mon, 04 Aug 2014 17:05:10 GMT

This is actually a summary of few talks which deals with "under the hood" topics, the talks partially overlap. Those topics include - memory allocation and management, inheritance, over-ridden built-in methods, etc.

Relevant talks -

The magic of attribute access by Petr Viktorin

https://ep2014.europython.eu/en/schedule/sessions/123/

Performance Python for Numerical Algorithms by Yves

https://ep2014.europython.eu/en/schedule/sessions/64/

Metaprogramming, from Decorators to Macros by Andrea Crotti

https://ep2014.europython.eu/en/schedule/sessions/84/

Everything you always wanted to know about Memory in Python but were afraid to ask by Piotr Przymus

https://ep2014.europython.eu/en/schedule/sessions/28/

Practical summary -

__slots__ argument - limited the memory allocation for objects in Python by overriding the __dict__ attribute.
Strings - the empty strings and strings of length=1 are saved as constants. Use intern (or sys.intern in python 3.x) on strings to avoid allocating string variables with the same values this will help making memory usage more efficiency and quicker string comparison. More about this topic here .
Numerical algorithms - the way a 2-dimensional array is allocated (row wise or column wise) and store has a great impact on the performance of different algorithms (even for simple sum function).
Working with the GPU - there are packages that process some of the data on the GPU. It is efficient when the data is big and less efficient when the data is small since copying all the data to the GPU has some overhead.
Cython use c "malloc" function for re-allocating space when list \ dictionaries \ set grow or shrink. On one hand this function can be overridden, on the other hand one can try to avoid costly processes which cause space allocation operations or to use more efficient data structures, e.g list instead of dictionary where it is possible.
Note the garbage collector! Python garbage collector is based on reference count. Over-ridding the __dell__ function may disrupt the garbage collector.
Suggested Profiling and monitoring tools - psutil, memory_profiler, objgraph, RunSnakeRun + Meliae, valgrind

Bottom line of all of those - "knowledge itself is power". I.e. knowing the internal and the impact of what we are doing can bring to a significant improvements.

There are always several ways to do things and each has cons and pros fitted to the specific case. Some of those are simple to implement and use and can donate to a great improvement on both running time and memory usage. On the other hand some of those suggestion are really "shoot in the foot" - causing memory leaks and other unexpected behavior, beware.

5 interesting things (03/08/2014)

Sun, 03 Aug 2014 17:07:05 GMT

Goooooooooaaaaaalllll - world cup is over but Python and soccer are forever :). This post tries to identify goals and interesting events on soccer games based on the volume on youtube. This post only touches the subject but what a nice start!

http://zulko.github.io/blog/2014/07/04/automatic-soccer-highlights-compilations-with-python/

Scheduling with Celery - and I thought I can only make a soup with celery.. Scheduling is not the main goal of Celery but it can be used as such, cron style. More about it -

http://www.caktusgroup.com/blog/2014/06/23/scheduling-tasks-celery/

Markov chains - the two links below complement one another. One has a great visualization and the other explain things a bit more deeply.

http://setosa.io/blog/2014/07/26/markov-chains/

http://www.analyticsvidhya.com/blog/2014/07/markov-chain-simplified

My Ig-Noble candidate -

http://www.bbc.com/news/magazine-20578627

Hotels by WiFi - geeky but important those days.

http://www.hotelwifitest.com

Graph Databases, a little connected tour

Fri, 25 Jul 2014 13:56:19 GMT

EuroPython talk by: Francisco Fernández Castaño

https://ep2014.europython.eu/en/schedule/sessions/70/

The talk was classified as novice and so it was - very basic graph database ideas.

Castaño started by presenting the general idea of graph databases and showing some use cases.

One of the most known use cases - social media data, friends-of-friends.

Then he presented Neo4j which is graph database written in Java but originally was written in Python. Neo4j is known to be very scalable and to support ACID transactions. Another nice property about Neo4j is the ability to extend the give rest API but your own needs.

The next part of talked focused on the cypher language - which is the way to query the Neo4j database. Neo4j have some nice UI properties and I really missed some reference or example for that on this talk.

Last but not least tip given in the talk, you can try Neo4j without having to install it on your machine - http://www.graphenedb.com/ (free sand box of up to 1k nodes and 10k edges).

The Shogun Machine Learning Toolbox

Fri, 25 Jul 2014 13:37:17 GMT

EuroPython talk by: Heiko Strathmann, herrstrathmann.de

https://ep2014.europython.eu/en/schedule/sessions/103/

Strathmann presented the Shogun toolbox. Shogun is a Machine Learning toolbox which implement most of the common supervised and unsupervised algorithms. Shogun is implemented in C++ and interacts with Python as well as with Java, Ruby, Matlab, Lau, etc. Shogun is meant to run on single machine and no distributed.

Shogun is an open source project started on 2004. It has 8 core developers + 20 contributors and it is fairly active. It also has a collebaration with Google summer of code - 29 projects so far, 8 on going.

The C++ maybe the most significant advantage of Shogun over SciPy. It is simply faster and enables more efficient memory allocation and run time optimization and well as multi-language support . Therefore enabling training and classifying data based on larger number of samples and higher dimension.

The binding \ interfaces to other languages is done using - http://www.swig.org.

Building, bug tracing and deployment is done using - http://buildbot.net/.

Summarizing - Specially when going larger Shogun seems like a good alternative to SciPy (competition makes both products better). Shogun site offers some tutorials include notebooks and demos.

Learning Chess from Data

Thu, 24 Jul 2014 19:26:12 GMT

EuroPython talk I gave today. Joint work with N.M.

Talk is here - https://ep2014.europython.eu/en/schedule/sessions/18/

Slides and code - https://github.com/nivm/learningchess

Log everything with logstash and elasticsearch

Tue, 22 Jul 2014 18:38:38 GMT

EuroPython talk by: Peter Hoffmann, @peterhoffmann

https://ep2014.europython.eu/en/schedule/sessions/47/

A talk by Peter Hoffmann from Blue Yonder which is one of the sponsors of the conference.

Another very good talk by an experienced speaker. However the name is kind of misleading. Yes - logstash and elasticsearch were mentioned however to main concept of the talk was really logging chain and centralized logging while logstash and elasticsearch are two tools along this chain and there are some alternatives in every step of this chain.

This talk gave me a lot to think about the logging on the company aspect and how should they run, monitor, etc. Also there is some tension to solve \ define about what is logged and what error message \ outputs an app should provide ("logging best practices").

The video is already available in the EuroPython site and I hope that the slides would be available too soon.

Full Stack Python

Tue, 22 Jul 2014 17:53:20 GMT

EuroPython talk by: Matt Makai, @mattmakai

https://ep2014.europython.eu/en/schedule/sessions/41/

I expected this talk to be full with buzzwords and it was... but in a good sense.

Makai is the builder of Full Stack Python site as such he spoke about what you need from the moment you have an idea to a python web app until you deploy including all the essential steps - wsgi server, hosting, logging etc.

Every layer in the site include relevant link and tutorials. Good starting point for a python web-app developer.

The talk was interesting due to Makai enthusiasm to teach and to share his knowledge (and of course to promote his site and that's legit as well) and his professional knowledge. There are really few people the can speak so fluently for 25 minutes.

Slides - http://www.mattmakai.com/presentations/2014-full-stack-python-berlin.html

5 interesting things (18/7/2014)

Fri, 18 Jul 2014 09:14:54 GMT

A\B Testing - No need to say that A\B is a very hot buzz word in the industry. A\B tests were used long ago in psychology but today there are much more accessible and easy to set. The following series of posts (one is still not publish) describe 5 simple guide lines for A\B testing. http://sl8r000.github.io/ab_testing_statistics/ Shellify - decorator which turns Python models into shell. I don't see an everyday usage to it, maybe on developing but it has a high coolness factor which is important as well. https://bitbucket.org/johannestaas/shellify Code review without your glasses - I like this post because in my humble opinion it is very creative, beyond the box. It is also a good reminder about about what to notice in code review. http://robertheaton.com/2014/06/20/code-review-without-your-eyes/ Side kick - the last link reminded me of rubber duck debugging and on the way I found this - http://rubberduckreview.com/ Deployment academy - a series of posts by Rainforest in their blog. This time a post about "zero downtime database migrations". The post is simple and easy to understand and answer a common issues in the life of a developer. Of course, practices and specific problems differ from one organization to another but the core ideas are as is. https://blog.rainforestqa.com/2014-06-27-zero-downtime-database-migrations/

Awesome *

Thu, 03 Jul 2014 14:23:38 GMT

Are we going back?

In the past week I have bumped into two github repositories such as awesome-python and awesome-sysadmin. Both repositories do a great job and compose and interesting list \ index of relevant tools.

However, this made me feel that we are going back. Those indices reminded me the pre-search-engines days or shell I say the BME days (before modern era ;). Specifically this reminded me of Alta Vista (but also other indices sites - do you recall Lycos?) where all the links where indexed under some category and sub categories and one should have dig in those categories to find what he looked for.

Aren't the search engines today strong enough to answer the query "python machine learning package"?. Is human indexing really our resort? I don't really think so. I believe in the power of the human behind the machine and their ability to build a good enough searching engines. Those indices can be very useful but I would rather have them built automatically and not manually.

5 interesting things (23/6/2014)

Mon, 23 Jun 2014 18:35:21 GMT

Celery best practices - python celery package is on my "todo list" once I have a relevant task. However, some of the thoughts and ideas which are talked about in this post are more general regarding software engineering.

https://denibertovic.com/posts/celery-best-practices/

Keep calm and learn d3.js - d3.js is also something which is on my "todo list". I used it randomly here-and-there but would like to do it better (and also to gain some deeper experience with JS). However, this is a very nice tutorial to start with d3.js.

http://slides.com/kentenglish/getting-started-with-d3-js-using-the-toronto-parking-ticket-data

DBSCAN well explained - unsupervised learning sometimes feels like no man's land and k-means is almost always the choice when picking a clustering algorithm. This post not only does a great job by explaining the algorithm itself but it also gives great examples and show how to adjust the parameters.

http://cjauvin.blogspot.ca/2014/06/dbscan-blues.html

Machine learning in Airbnb - This post is a case study of machine learning as it is used in Airbnb. I like reading such posts because it is interesting to learn about the challenges other organizations face and about their solutions and how they use existing tools and packages and adjust and optimize them to their need (in this post - scikit learn and re-writing R export function in c++).

http://nerds.airbnb.com/architecting-machine-learning-system-risk/

Current events - world cup predictions - as I love both sports and machine learning, statistics and so forth there are several posts which try to employ ML techniques to predict world cup results. The most interesting post I read so far is Nate Silver's post. It takes many features into account and explain them clearly.

http://fivethirtyeight.com/features/its-brazils-world-cup-to-lose/

Andrew Ng and PyStruct meetup

Tue, 17 Jun 2014 09:33:23 GMT

Yesterday I attended the "Andrew Ng and pyStruct" meetup.

http://www.meetup.com/berlin-machine-learning/events/179264562/

I was lucky enough to get a place to the meetup due to the Germany-Portugal game that happened on the same time :)

The first part by Andrew Ng was a video meetup joint to 3 locations - Paris, Zurich and Berlin. Andrew Ng is a co-founder of Coursera and a Machine Learning guru. He teaches the ML course in Coursera which is one of the most popular courses in Coursera (took it myself and it is a very good and structured introduction to machine learning, new session started yesterday). He teaches in Standford and soon he will be leaving to Baidu research.

The talk included 15-20 minutes of introduction to deep learning, recent results, applications and challenges. He mainly focused on scaling up deep learning algorithms for using billions features \ properties. The rest of the talk was question answering mostly regarding the theoretical aspects of deep learning, future challenges, etc. For me one of the most important things he said was "innovation is a result of team work".

Some known applications of deep learning is - speak recognition, image processing, etc.

In the end he suggested taking Stanford deep learning tutorial - http://deeplearning.stanford.edu/tutorial/

T.R - there are currently 2 python packages I know which deal with deep learning -

Theano
PyLearn2 - mostly based on Theano

The next talk was given by Andreas Mueller. You can find his slides here.

Muelller introduced structured prediction which is a natural extension or a generalization of regression problems to a structured output rather than just a number \ class . Structured learning has advantage over other algorithms of supervised learning as it can learn several properties at once and use the correlations between those properties.

Example - costumers data, several properties of costumers - gender, marriage status, has children, owns a car, etc. One can guess that married and has children properties are highly correlated and that when learning about those properties together there is a better chance of getting good results. It is better than LDA in the sense that it has less classes (not every combination of the variables is a class) and it requires less training data.

Other examples include pixel classification - classifying each pixel to an object in the image and OCR, etc.

He then talked about PyStruct - a python package for structured prediction. Actually not much to add that is not written in the documents.

5 interesting things (14/6/2014)

Sat, 14 Jun 2014 15:30:13 GMT

My little helper - search engine for code examples and use cases. I haven't yet tried that "live" when I needed something, but I hope I will remember it next time.

https://sourcegraph.com/

Httpie and Percol - two unrelated tools but I see a lot of similarity between them as they try to change the common way we do things on command line. Http try to make curl requests more human understandable and percol which try to make filtering using piping more interactive. Reminds me a bit of edinting in sublime.

https://github.com/jakubroztocil/httpie

https://github.com/mooz/percol

Python 3 is good for you - A bit long but very interesting. Overview 10 features which are new in Python 3. There were recently a lot of posts around the web discussing whether Python 3 is better than Python 2.x, whether Python 3 should be rolled back and buried forever etc. This is one of the most informative posts (although it could be summarized and shorter) I have read. I think that one of the main reasons organization don't currently move to Python 3 beside the fact the people and organization don't love changes is because it is an expensive process (mainly compatibility) and even this post does not succeed in convincing with its' added value.

http://asmeurer.github.io/python3-presentation/slides.html#1

Are we humans or are we dancers, sorry computers -

http://wired.com/2014/01/how-to-hack-okcupid/all

Kernel tricks - very clear post about kernel trick which also make clear additional machine learning terminology and the examples are very good. I would say that this is a very good post for beginners-intermediates in Machine Learning. Going the extra mile would be writing something similar about PCA has it has a lot of similar ideas.

http://www.eric-kim.net/eric-kim-net/posts/1/kernel_trick.html

Visualization - Data scientist toolkit

Tue, 03 Jun 2014 10:16:03 GMT

Data scientist are said to have better development knowledge than the average statistician and better statistic knowledge than the average developer. However, together with those skills one also needs marketing skills - the ability to communicate your, no so simple job and results to other people. Those people can be the CTO or VP R&D, team members, customers or sales and marketing people. They don't necessarily share your knowledge or dive into the details as fast as you.

One of the best ways to make data and results accessible is creating visualizations, automatically of course. In this post I'll review several visualizations tools, mostly for Python with some additional side kicks.

Matplotlib - probably the most known python visualization package. Includes most of the standard charts - bar charts, pie charts, scatters, ability to embed images, etc. Since there are many users using it there are many questions, examples and documentations around the web. However, the downside for me is that it is more complex than it should be. I have used it in several projects and I don't yet acquired the intuition to fully utilize. Matplotlib have several extensions including -

mplot3d - 3d visualization
cartopy - for maps charts

graphviz - Designated for drawing graphs. Graph drawing software with python package. pygraphviz is a python package for graphviz which provides a drawing layer and graph layout algorithms. The first downside of this is that you need to download the graphviz software. I have done it several times on several different machines (most of the consist of ubuntu) it never passed smoothly and I was not able to do it only from the command line which make it problematic if one wants to deploy it on remote machines. I believe that it could be done but at the moment I find this process only as an irksome overhead.

Side kicks -

PyDot - Implements DOT graph description language. PyDot is basically an interface to interact with PyGraphviz dot layout. The main advantage of the dot files and data is the advantage in standardization - one can create dot file in one process and use it in other process. DOT is an intuitive language which focuses on drawing the graph and not on calculating the graph. I would say that it is the last step in the chain.
Networkx - a package for working and manipulating graph. Implements many graph algorithms such as shortest path, clustering, minimum spanning tree, etc. The graphs created in Networkx can be drawn using either matplotlib or pygraphviz and can also create dot files.

Vincent - A relatively new python visualization package. Vincent translates Python to Vega which is a visualization grammar. I like it because it is easy, interactive and simple to output either as JSON or as HTML . However, I'm not sure that both Vincent and Vega are mature enough at this point to answer all the needs. It is important to mention that Vega is actually a wrapper above D3 which is an amazing tool with growing community.

Additional related tools I'm not (yet) experienced with -

xlsxwriter - creating excel files (xlsx format) including embedding charts on those files.
plot.ly - very talked about tool for collaborating data and graphing tool which have a Python client. I try to keep my data as private as possible and don't want to be dependent on internet connection (for example - creating graph with a lot of data) so this is the down side for me in this tool. However, the social \ collaborative aspect of this product is also an important part and the graphing is only one aspect of it.
Google charts - same downside as plot.ly - I like to be as independent as possible. However, comparing to plot.ly it looks more mature and has far more options, chart types than plot.ly at this stage and there is also a sand box to play with it. Plot.ly has advantages over Google charts in the ease of usage for non programmers.
Bokeh - Nice, interactive charts on large data sets. Maybe the next big thing for plotting in Python.

5 interesting things (2/6/2014)

Mon, 02 Jun 2014 10:11:38 GMT

Tell me your name and I'll tell you your age - playing with data and a conditioned distribution

http://fivethirtyeight.com/features/how-to-tell-someones-age-when-all-you-know-is-her-name/

Analysis of Over 2,000 Computer Science Professors at Top Universities - while this data is interesting I would say that it is only an introduction. It is not an anthropological study but it is still curious to ask how many of those professors are women? what are their ages? is the median age of a women when she joins similar to those of man? What is the distribution over the different sub-fields (i.e. are women more to theory than man?)? What is the percentage of the foreign professor or professor who did not took their undergrad studies in the US?

This is a nice starting point and there are many other things that can be done with this data.

http://jeffhuang.com/computer_science_professors.html

Good runner, bad investor? - An empirical evidence \ variation on Kahneman and Tversky "Cognitive Biases" but a nice one.

http://www.nytimes.com/2014/04/23/upshot/what-good-marathons-and-bad-investments-have-in-common.html

Link shortening - short, elegant links satisfy both technological standards (twitter 140 char limit) and human needs for short links which fit in line. Apparently those links cause much overhead due to the redirection, harm user experience and worth a lot of money.

http://t37.net/why-link-shorteners-harm-your-readers-and-destroy-the-web.html

Hacker News aggregation - I'm probably the last one to find out but for me it is an easier way to follow new links in Hacker News.

http://www.hndigest.com/

5 interesting things (23/5/2014)

Fri, 23 May 2014 10:02:09 GMT

Is that a baby bump in your status? Janet Vertesi tried to keep her pregnancy a way from the social networks. Did she succeed? What was the price? (high)

http://time.com/83200/privacy-internet-big-data-opt-out/

Spurious Correlation - "marriages in Alabama are causing deaths by electrocution" - the nightmare of every data scientist. Are two variable correlated? Is one the cause of the other or the other is the cause of the one? This post took it one step forward -

http://www.popsci.com/article/science/algorithm-reveals-link-between-sour-cream-and-traffic-accidents

History of Machine Learning - Another way to view machine learning and the progress made over the years. I'm wondering about the exact starting point \ initial formation of machine learning, I think it is a bit earlier than what is stated there. The reason to read this article is given in Pirkei Avot: "Know from where you came and where you are going and before whom you are destined to give account and reckoning" (3;1).

www.erogol.com/brief-history-machine-learning/

Breaking Python 2 for 1 - I have missed the first post when it was published so got 2 for 1 now. Two fascinating blog posts about under the hood of CPython, ctypes, garbage collector, etc. I really appreciate the hands on approach, it reveals a side of Python which I'm not exposed to on my daily work (and probably for most projects it is not a good practice to mess with).

http://blog.hakril.net/articles/0-understanding-python-by-breaking-it.html

http://blog.hakril.net/articles/1-understanding-python-by-breaking-it---lvl-2.html

Where is Waldo? - find Waldo in just 42 lines of code (not surprising having that 42 is the meaning of life). I have not background in image processing but those tools really make it easy, almost out of the box (and great flashback to my childhood).

http://machinelearningmastery.com/using-opencv-python-and-template-matching-to-play-wheres-waldo/

Learning: Spatial database

Sun, 18 May 2014 14:13:33 GMT

A while ago I was asked a question about Spatial database and figured out I didn't know anything about it. So, I use this platform to document my learning about this topic.

What is Spatial data?

The first lesson in "database 101" is that the design of a database should fit to the queries made on it. The special thing about spatial databases is that they hold data about objects in a geometric spaces. Most of the traditional databases don't index the information in an optimize way for spatial queries, examples for such queries can be -

How to far is location A to location B?
How to get from location A to location B?
What is the closest restaurants to some location?
What is the path people usualy take in some park \ market?
Is location A inside city B?

We can divide spatial queries to few groups (based on Wikipedia)

Spatial Measurements: Computes line length, polygon area, the distance between geometries, etc.
Spatial Functions: Modify existing features to create new ones, for example by providing a buffer around them, intersecting features, etc.
Spatial Predicates: Allows true/false queries about spatial relationships between geometries. Examples include "do two polygons overlap" or 'is there a residence located within a mile of the area we are planning to build the landfill?'
Geometry Constructors: Creates new geometries, usually by specifying the vertices (points or nodes) which define the shape.
Observer Functions: Queries which return specific information about a feature such as the location of the center of a circle.

What is next?

This post is actually the preview for additional two posts - a theoretic one and a productive one.

The theoretic one will compare the different spatial indices and on which needs each of them answers. This post will talk about the most common index of this kind is R-Tree and some of its' extensions and presumably some additional related theory.

The productive one will present some of the current solutions for spatial data among them is PostGis used in PostgreSQL, MySQL and additional NoSQL solution (probably either Neo4j or MongoDB).

Beside the basic curiosity of learning a new thing I believe that the field of "Location intelligence", i.e., using and analyzing spatial information as part of the decision making process, is an emerging field yet to be discovered. In other words location intelligence is another layer of business intelligence (BI) and as the technology evolves and the data is gathered we can now use this data better than ever before.

Some of the current commercial application of location intelligence -

Route planning - such as Waze
Geo targeting - for example using IP address to display relevant ads.
Travel planning - hotel, restaurant, attractions such as GetYourGuide
Sales analysis - such as SpatialKey

5 interesting things (16/5/2014)

Fri, 16 May 2014 15:32:36 GMT

MailPin -Turn an email to a web page - For me it is a cool tool that I probably won't use but it is cool and that is also important thing those days.
http://mailp.in/

10 years to LiMux project - "How Munich switched 15,000 PCS from Windows to Linux". Interesting both on the technological aspect and both on the sociological \ anthropological aspect.

http://www.linuxvoice.com/the-big-switch/

Why Python is Slow: Looking Under the Hood: I really like posts which help me understand better what I'm doing and this one is also very well written. Through Python Weekly.

http://jakevdp.github.io/blog/2014/05/09/why-python-is-slow/

How to Marry The Right Girl: A Mathematical Solution: I must admit that I don't like those kind of posts or at least their header. I feel they are very sexist not to say misanthropic. However, there was one point which I relate to in this post but on a different scope work interviews. Doing some work interviews lately, my gut feeling is that being in the first group really harm the chances of getting hired. Many interviewers don't really know what exactly they are looking for and refine their requirements only after several interviews. To conclude the content was interesting but could be written in a different tone.

http://www.npr.org/blogs/krulwich/2014/05/15/312537965/how-to-marry-the-right-girl-a-mathematical-solution

The Economics of Kickstart project: Crowd-funding is one of the latest trends. As in everything there are both advantages and disadvantages. One of the main advantages as I see it is the global exposure to one's ideas. The down side of this advantage is that it eases the way to still ideas (some my call it inspiration). More over it does not really clearly if it is always profitable as expected. More about it in the attached post.

https://medium.com/@cameronmoll/5b374cfeb43c

Coursera R programming course

Mon, 12 May 2014 08:52:23 GMT

I have recently took "R programming" course in Coursera. This is the second course I take there. Before that I took "Machine Learning" course which was much heavier course with respect both to what was taught, to the course length and to the assignments and other requirements.

In the world of SciPy, Numpy, Pandas and others in Python I don't really see the advantage in R. Those libraries have almost the same capabilities while Python is a much stronger and more common and therefore supported and documented tool.
Visualizing the data - for me one of the best practices to gain better understanding of the data and to introduce the data to other people is creating a visualization. I really lack that part in Coursera course, I think that they should have done that extra mile, this would make the data processing more meaningful.
I'm not an R expert now, not even close. However, I gained some background next time I will need to handle code in R or that I will need to consider using R in a project. I saw some similarity to Matlab syntax so this knowledge might be relevant there too.
Language background, advantages, disadvantages, limits, theory (scoping, typing) etc. is important yet asking in a quiz from which university the language developers came from is not that interesting.

This is my code repository for this course, not brilliant implementation but those the job - https://github.com/tomron/rprog