JavaScript Fuzzing in Mozilla, 2017

JavaScript Fuzzing in Mozilla, 2017

Agenda

  • What is fuzzing?
  • Why is fuzzing successful?
  • Tips for effective fuzzing

Fuzzing

xkcd compiling
xkcd fuzzing

Quote

the people writing and running fuzzers are by far the most effective QA people at Mozilla.

What is fuzzing?

  • Give random input to software
  • May find easy-to-detect problems
    • Crashes
    • Assertion failures
  • These can manifest in the real-world

Dynamic analysis tools

  1. Address Sanitizer (ASan)
  2. Thread Sanitizer (TSan)
  3. Leak Sanitizer (LSan)
  4. Valgrind
  5. and Undefined Behaviour Sanitizer (UBSan), etc.

Platform Fuzzing team

Map of fuzzing team

List of fuzzing tools

List of fuzzing tools (cont'd)

Bug Bounty programs

Bug bounty programs

Why is fuzzing successful?

  • Bug bounty programs
  • Bug counts
    • ~7,000 - jsfunfuzz & DOMFuzz
    • >10% sec-critical

Why is fuzzing successful? (cont'd)

Possible focus areas for sec bugs

  • In an unsafe language?
  • Well-sandboxed or not?
  • Parsing controllable input?
  • Security boundaries?
  • Areas other security researchers are working on?

Good starting points for fuzzing

  • Isolated modules
    • SpiderMonkey JS shell
    • media shell
  • Can be tested on Firefox Desktop

Typical issues a large project faces

  • Bugs
  • Tests
  • Manpower
  • Resistance
  • Hard to find serious bugs

Case study: JavaScript team

  • Too many fuzzbugs → resistance
  • Value
  • Fuzzing team → QA
  • Took time to realize

Case study: JavaScript team (cont'd)

  • Fuzzbug reports
    • Concise
    • Popular
    • May be duplicates of real-world bugs
    • Devs sometimes wait for fuzzbugs

Quote

Fuzzing is an effective way to find corner cases overlooked when creating a patch and a test for it. A comprehensive fuzz bug report can also provide more information on how to fix it.

Working as a team

  • Fix fuzzbugs
  • Tests land on trunk (master branch)
  • More starting points for fuzzing

Devs code to help find more bugs

  • Add assertions & testing functions
  • Test components in isolation
  • Dynamic analysis tool support
    • AddressSanitizer (ASan)
    • ThreadSanitizer (TSan)
    • Valgrind

Results

  • Security bugs → higher-priority
  • Fuzzblockers → higher-priority
  • Fuzzbugs → generally lower-priority
  • The more we discover, the less others find
  • More bugs discovered, as we scale

Potential pitfalls

  • Automation frameworks
  • Devices not powerful enough
    • Mitigation: ARM simulator binary

Factors for success

  • Patches first land on trunk (master branch)
  • Backout when regressions occur
  • Fuzzblockers are fixed
  • Tests are rarely disabled

Suggested automation building blocks

  • Input creation (the fuzzer)
  • Instrumentation
  • Monitoring
  • Log bucketing
  • Testcase reduction
  • Bisection

Mozilla Security GitHub projects

  • funfuzz (Gary Kwong)
    • jsfunfuzz harness
  • FuzzManager (Christian Holler) & EC2SpotManager
    • Fuzzing management tools collection
  • Laniakea (Christoph Diehl)
    • Tool to manage EC2 instances at AWS

Mozilla Security GitHub projects (cont'd)

Q&A

Conclusion

Miscellaneous

Red panda (Firefox)