Counters using Cascading Flow Listeners in Scalding

As of now, Scalding doesn’t provide full support for counters – you will find a few pull requests and the Stats class, nothing more. This will probably change in the future, until then, I found using Cascading FlowListeners for counters was the most convenient solution.

In my Scalding Job class, I just need to override the listeners and add my own implementation of a FlowListener:

class MyJob (args: Args) extends Job(args) {

  // my stats group, this helps keep them apart from others
  val statsGroupName = "MyStatsGroup"
  // my metric
  val deletedUsersStat = Stat("deletedUsers", statsGroupName)

  override def listeners = super.listeners ++ List(new FlowListener {

    override def onStarting(flow: Flow[_]) {
       // init stuff goes in here

    override def onCompleted(flow: Flow[_]) {
      try {
        val fs = flow.getFlowStats
        fs.getCounterGroups.foreach { group =>
          fs.getCountersFor(group).foreach { counter =>
            println(group + "::" + counter + ":" + fs.getCounterValue(group, counter))
        val myStat = (fs.getCounterValue(,
      } catch {
        case e: Exception => e.printStackTrace()

    override def onStopping(flow: Flow[_]) {


    override def onThrowable(flow: Flow[_], e: Throwable): Boolean = {
      // return true after handling, otherwise your listener will stop

You will have to use the reference to your Stat object to update and increment your metric:

  val {
     //... other map code

deletedUsersStat being the stat val declared at the top of MyJob in the code block with the listener declaration

This entry was posted in Uncategorized and tagged , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s