Counters using Cascading Flow Listeners in Scalding

As of now, Scalding doesn’t provide full support for counters – you will find a few pull requests and the Stats class, nothing more. This will probably change in the future, until then, I found using Cascading FlowListeners for counters was the most convenient solution.

In my Scalding Job class, I just need to override the listeners and add my own implementation of a FlowListener:


class MyJob (args: Args) extends Job(args) {

  // my stats group, this helps keep them apart from others
  val statsGroupName = "MyStatsGroup"
  // my metric
  val deletedUsersStat = Stat("deletedUsers", statsGroupName)


  override def listeners = super.listeners ++ List(new FlowListener {

    override def onStarting(flow: Flow[_]) {
       // init stuff goes in here
    }

    override def onCompleted(flow: Flow[_]) {
      try {
        val fs = flow.getFlowStats
        fs.getCounterGroups.foreach { group =>
          fs.getCountersFor(group).foreach { counter =>
            println(group + "::" + counter + ":" + fs.getCounterValue(group, counter))
          }
        }
        val myStat = (fs.getCounterValue(deletedUsersStat.group, deletedUsersStat.name))
      } catch {
        case e: Exception => e.printStackTrace()
      }
    }

    override def onStopping(flow: Flow[_]) {

    }

    override def onThrowable(flow: Flow[_], e: Throwable): Boolean = {
      e.printStackTrace()
      // return true after handling, otherwise your listener will stop
      true
    }
  })

You will have to use the reference to your Stat object to update and increment your metric:


  val myPipe.map {
     //... other map code
     deletedUsersStat.incBy(1)
  }

deletedUsersStat being the stat val declared at the top of MyJob in the code block with the listener declaration

Advertisements
This entry was posted in Uncategorized and tagged , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s