10 SQL Tricks that You Didn't Think Were Possible

4. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 Why are you here? Because clickbait headlines work! … Trick #3 Changed work forever! … Trick #8 Makes doctors hate you! … Trick #9 is bigger than life!

5. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 Me – @lukaseder SQL is a device whose mystery is only exceeded by its power! - Founder and CEO at Data Geekery - Oracle Java Champion - JUG.ch Board Member

6. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 Why do I talk about SQL? SQL is the only ever successful, mainstream, and general- purpose 4GL (Fourth- Generation Programming Language) And it is awesome!

11. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 What is SQL? SQL is the original microservice Just install a single stored procedure in an Oracle XE instance, deploy, done.

13. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 Who thinks this is SQL? @Entity @Table(name = "EVENTS") public class Event { private Long id; private String title; private Date date; @Id @GeneratedValue(generator = "increment") @GenericGenerator(name = "increment", strategy = "increment") public Long getId() { /* … */ } @Temporal(TemporalType.TIMESTAMP) @Column(name = "EVENT_DATE") public Date getDate() { /* … */ }

14. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 Or this…? @OneToMany(mappedBy = "destCustomerId") @ManyToMany @Fetch(FetchMode.SUBSELECT) @JoinTable( name = "customer_dealer_map", joinColumns = { @JoinColumn(name = "customer_id", referencedColumnName = "id") }, inverseJoinColumns = { @JoinColumn(name = "dealer_id", referencedColumnName = "id") } ) private Collection dealers; Found at http://stackoverflow.com/q/17491912/521799

17. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 This is also SQL -- Query from http://explainextended.com/2013/12/31/happy-new-year-5/ WITH RECURSIVE q(r, i, rx, ix, g) AS ( SELECT r::DOUBLE PRECISION * 0.02, i::DOUBLE PRECISION * 0.02, .0::DOUBLE PRECISION , .0::DOUBLE PRECISION, 0 FROM generate_series(-60, 20) r, generate_series(-50, 50) i UNION ALL SELECT r, i, CASE WHEN abs(rx * rx + ix * ix) <= 2 THEN rx * rx - ix * ix END + r, CASE WHEN abs(rx * rx + ix * ix) <= 2 THEN 2 * rx * ix END + i, g + 1 FROM q WHERE rx IS NOT NULL AND g < 99 ) SELECT array_to_string(array_agg(s ORDER BY r), '') FROM ( SELECT i, r, substring(' .:-=+*#%@', max(g) / 10 + 1, 1) s FROM q GROUP BY i, r ) q GROUP BY i ORDER BY i

18. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 This is also SQL: Generating the Mandelbrot Set -- Query from http://explainextended.com/2013/12/31/happy-new-year-5/ WITH RECURSIVE q(r, i, rx, ix, g) AS ( SELECT r::DOUBLE PRECISION * 0.02, i::DOUBLE PRECISION * 0.02, .0::DOUBLE PRECISION , .0::DOUBLE PRECISION, 0 FROM generate_series(-60, 20) r, generate_series(-50, 50) i UNION ALL SELECT r, i, CASE WHEN abs(rx * rx + ix * ix) <= 2 THEN rx * rx - ix * ix END + r, CASE WHEN abs(rx * rx + ix * ix) <= 2 THEN 2 * rx * ix END + i, g + 1 FROM q WHERE rx IS NOT NULL AND g < 99 ) SELECT array_to_string(array_agg(s ORDER BY r), '') FROM ( SELECT i, r, substring(' .:-=+*#%@', max(g) / 10 + 1, 1) s FROM q GROUP BY i, r ) q GROUP BY i ORDER BY i

23. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 Which do you feel is more awesome? This? Siri, what is the meaning of life? Image credit: https://www.flickr.com/photos/procsilas/12821454664 By Procsilas Moscas. License CC-BY 2.0

24. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 Which do you feel is more awesome? Or this? Image credit: https://www.flickr.com/photos/ajmexico/3281139507 By ajmexico. License CC-BY 2.0

27. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 What’s the problem with SQL? – SQL code WITH RECURSIVE t(d) AS ( SELECT DATE '2005-07-01' UNION ALL SELECT (d + INTERVAL '1 days')::DATE FROM t WHERE d < DATE '2005-07-31' ) SELECT * FROM t

28. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 What’s the problem with SQL? – COBOL code DATA DIVISION. FILE SECTION. FD Sales-File. 01 Sales-Rec. 88 End-Of-Sales-File VALUE HIGH-VALUES. 02 SF-Cust-Id PIC X(5). 02 SF-Cust-Name PIC X(20). 02 SF-Oil-Id. 03 FILLER PIC X. 88 Essential-Oil VALUE "1". 03 SF-Oil-Name PIC 99. 02 SF-Unit-Size PIC 99. 02 SF-Units-Sold PIC 999.

36. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 Remember this from this talk: The SQL muscle Image credit: https://www.flickr.com/photos/flamephoenix1991/8376271918 By _DJ_. License CC-BY SA 2.0

37. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 Remember this from this talk: The SQL muscle Image credit: https://www.flickr.com/photos/flamephoenix1991/8376271918 By _DJ_. License CC-BY SA 2.0 This is the SQL muscle. It needs constant training and practice

39. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 10 SQL tricks to convince you SQL is awesome 1. Everything is a table 2. Data generation with recursive SQL 3. Running total calculations 4. Finding the length of a series 5. Finding the largest series with no gaps 6. The subset sum problem with SQL 7. Capping a running total 8. Time series pattern recognition 9. Pivoting and unpivoting 10.Abusing XML and JSON (don’t do this at home)

43. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 Speaking of slides: Let’s thank our patron saint Ada Lovelace Without her, instead of writing SQL, we would all be writing Powerpoint or something

48. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 1. Everything is a table SELECT * FROM ( -- "values constructor" VALUES (1, 'a'), (2, 'b') ) t(a, b) -- "derived column list" But did you know this?

50. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 1. Everything is a table SELECT * FROM ( SELECT 1 AS a, 'a' AS b FROM dual UNION ALL SELECT 2, 'b' FROM dual ) t But did you know this?

52. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 1. Everything is a table – Compare it to Java 8 TABLE : Stream<Tuple<..>> SELECT : map() DISTINCT : distinct() JOIN : flatMap() WHERE / HAVING : filter() GROUP BY : collect() ORDER BY : sorted() UNION ALL : concat() See: http://blog.jooq.org/2015/08/13/common-sql-clauses-and-their-equivalents-in-java-8-streams/

54. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 1. Everything is a table – Compare it to Java 8 Seq.seq(persons) .collect( count(), max(Person::getAge), min(Person::getHeight), avg(Person::getWeight) ); // (3, Optional[35], // Optional[1.69], Optional[70.0])

56. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 2. Data Generation with Recursive SQL -- Table variables WITH t1(v1, v2) AS (SELECT 1, 2), t2(w1, w2) AS ( SELECT v1 * 2, v2 * 2 FROM t1 ) SELECT * FROM t1, t2

57. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 2. Data Generation with Recursive SQL WITH RECURSIVE t(v) AS ( SELECT 1 -- Seed Row UNION ALL SELECT v + 1 -- Recursion FROM t ) SELECT v FROM t LIMIT 5

61. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 2. Data Generation with Recursive SQL WITH RECURSIVE t(v) AS ( SELECT 1 -- Seed Row UNION ALL SELECT v + 1 -- Recursion FROM t ) SELECT v FROM t LIMIT 5 Credits for this lame Powerpoint joke: Hadi Hariri from JetBrains

64. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 Remember? -- Query from http://explainextended.com/2013/12/31/happy-new-year-5/ WITH RECURSIVE q(r, i, rx, ix, g) AS ( SELECT r::DOUBLE PRECISION * 0.02, i::DOUBLE PRECISION * 0.02, .0::DOUBLE PRECISION , .0::DOUBLE PRECISION, 0 FROM generate_series(-60, 20) r, generate_series(-50, 50) i UNION ALL SELECT r, i, CASE WHEN abs(rx * rx + ix * ix) <= 2 THEN rx * rx - ix * ix END + r, CASE WHEN abs(rx * rx + ix * ix) <= 2 THEN 2 * rx * ix END + i, g + 1 FROM q WHERE rx IS NOT NULL AND g < 99 ) SELECT array_to_string(array_agg(s ORDER BY r), '') FROM ( SELECT i, r, substring(' .:-=+*#%@', max(g) / 10 + 1, 1) s FROM q GROUP BY i, r ) q GROUP BY i ORDER BY i

65. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 2. Data Generation with Recursive SQL Applications: 1. Iterate from 1 to 10 2. Generate all dates in July 2016 3. Generating graphs (stay tuned!)

66. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 3. Running total calculations What is a running total? Ask your project manager to give you a crash course about the awesome Microsoft Excel!

67. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 3. Running total calculations What is a running total? Ask your project manager to give you a crash course about the awesome Microsoft Excel!

68. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 3. Running total calculations But first, a little theory about window functions There was SQL before window functions and there was SQL after window functions.

69. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 3. Running total calculations -- Aggregations / rankings on a subset of -- rows relative to the current row being -- transformed by SELECT function(...) OVER ( PARTITION BY ... ORDER BY ... ROWS BETWEEN ... AND ... ) What are window functions?

70. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 3. Running total calculations -- Aggregations / rankings on a subset of -- rows relative to the current row being -- transformed by SELECT function(...) OVER ( PARTITION BY length ORDER BY ... ROWS BETWEEN ... AND ... )

71. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 3. Running total calculations -- Aggregations / rankings on a subset of -- rows relative to the current row being -- transformed by SELECT function(...) OVER ( PARTITION BY length ORDER BY ... ROWS BETWEEN ... AND ... )

72. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 3. Running total calculations -- Aggregations / rankings on a subset of -- rows relative to the current row being -- transformed by SELECT function(...) OVER ( PARTITION BY ... ORDER BY title ROWS BETWEEN ... AND ... )

73. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 3. Running total calculations -- Aggregations / rankings on a subset of -- rows relative to the current row being -- transformed by SELECT function(...) OVER ( PARTITION BY ... ORDER BY title ROWS BETWEEN ... AND ... )

74. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 3. Running total calculations -- Aggregations / rankings on a subset of -- rows relative to the current row being -- transformed by SELECT row_number( ) OVER ( PARTITION BY ... ORDER BY title ROWS BETWEEN ... AND ... )

76. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 Let this settle a bit Image credit: https://www.flickr.com/photos/epler/482921404 By Jim Epler. License CC-BY 2.0 Window functions are aggregations / rankings on a subset of rows relative to the current row being transformed by SELECT

77. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 3. Running total calculations This is the data in the database table | ID | VALUE_DATE | AMOUNT | |------|------------|--------| | 9997 | 2014-03-18 | 99.17 | | 9981 | 2014-03-16 | 71.44 | | 9979 | 2014-03-16 | -94.60 | | 9977 | 2014-03-16 | -6.96 | | 9971 | 2014-03-15 | -65.95 |

78. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 3. Running total calculations This is what we want to calculate | ID | VALUE_DATE | AMOUNT | BALANCE | |------|------------|--------|------------| | 9997 | 2014-03-18 | 99.17 | 19985.81 | | 9981 | 2014-03-16 | 71.44 | 19886.64 | | 9979 | 2014-03-16 | -94.60 | 19815.20 | | 9977 | 2014-03-16 | -6.96 | 19909.80 | | 9971 | 2014-03-15 | -65.95 | 19916.76 |

79. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 3. Running total calculations This is how we calculate it | ID | VALUE_DATE | AMOUNT | BALANCE | |------|------------|--------|------------| | 9997 | 2014-03-18 -(99.17)| +19985.81 | | 9981 | 2014-03-16 -(71.44)| 19886.64 | | 9979 | 2014-03-16 -(-94.60)| 19815.20 | | 9977 | 2014-03-16 | -6.96 | =19909.80 | | 9971 | 2014-03-15 | -65.95 | 19916.76 |

80. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 SUM(t.amount) OVER ( PARTITION BY t.account_id ORDER BY t.value_date DESC, t.id DESC ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING )

84. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 Now we have the tool set. Remember these two advanced SQL features: 1. (Recursive) common table expressions 2. Window functions

89. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 4. Finding the largest series with no gaps | LOGIN_TIME | |---------------------| | 2014-03-18 05:37:13 | | 2014-03-16 08:31:47 | | 2014-03-16 06:11:17 | | 2014-03-16 05:59:33 | | 2014-03-15 11:17:28 | | 2014-03-15 10:00:11 | | 2014-03-15 07:45:27 | | 2014-03-15 07:42:19 | | 2014-03-14 09:38:12 |

90. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 4. Finding the largest series with no gaps | LOGIN_DATE | |------------| | 2014-03-18 | | 2014-03-16 | | 2014-03-15 | | 2014-03-14 |

91. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 Easy... 4. Finding the largest series with no gaps SELECT DISTINCT cast(login_time AS DATE) AS login_date FROM logins WHERE user_id = :user_id

92. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 4. Finding the largest series with no gaps | LOGIN_DATE | RN | |------------|----| | 2014-03-18 | 4 | | 2014-03-16 | 3 | | 2014-03-15 | 2 | | 2014-03-14 | 1 |

93. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 Still easy... 4. Finding the largest series with no gaps SELECT login_date, row_number() OVER (ORDER BY login_date) FROM login_dates

94. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 Now, what happens if we subtract...? 4. Finding the largest series with no gaps SELECT login_date - row_number() OVER (ORDER BY login_date) FROM login_dates

95. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 4. Finding the largest series with no gaps | LOGIN_DATE | RN | GRP | |------------|----|------------| | 2014-03-18 | 4 | 2014-03-14 | | 2014-03-16 | 3 | 2014-03-13 | | 2014-03-15 | 2 | 2014-03-13 | | 2014-03-14 | 1 | 2014-03-13 |

96. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 4. Finding the largest series with no gaps | LOGIN_DATE | RN | GRP | |------------|----|------------| | 2014-03-18 | 4 | 2014-03-14 | | 2014-03-16 | 3 | 2014-03-13 | | 2014-03-15 | 2 | 2014-03-13 | | 2014-03-14 | 1 | 2014-03-13 | Gap here Gap here

98. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 Easy explanation: 1. ROW_NUMBER() never has gaps 2. Our data, however, does 4. Finding the largest series with no gaps

99. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 So, just group by this difference! 4. Finding the largest series with no gaps SELECT min(login_date), max(login_date), max(login_date) - min(login_date) + 1 AS length FROM login_date_groups GROUP BY grp ORDER BY length DESC

100. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 4. Finding the largest series with no gaps | MIN | MAX | LENGTH | |------------|------------|--------| | 2014-03-14 | 2014-03-16 | 3 | | 2014-03-18 | 2014-03-18 | 1 |

101. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 4. Finding the largest series with no gaps WITH login_dates AS ( SELECT DISTINCT cast(login_time AS DATE) login_date FROM logins WHERE user_id = :user_id ), login_date_groups AS ( SELECT login_date, login_date - row_number() OVER (ORDER BY login_date) AS grp FROM login_dates ) SELECT min(login_date), max(login_date), max(login_date) - min(login_date) + 1 AS length FROM login_date_groups GROUP BY grp ORDER BY length DESC

103. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 5. Finding the length of a series | ID | VALUE_DATE | AMOUNT | |------|------------|--------| | 9997 | 2014-03-18 | 99.17 | | 9981 | 2014-03-16 | 71.44 | | 9979 | 2014-03-16 | -94.60 | | 9977 | 2014-03-16 | -6.96 | | 9971 | 2014-03-15 | -65.95 | | 9964 | 2014-03-15 | 15.13 | | 9962 | 2014-03-15 | 17.47 | | 9960 | 2014-03-15 | -3.55 | | 9959 | 2014-03-14 | 32.00 |

104. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 5. Finding the length of a series | ID | VALUE_DATE | AMOUNT | LENGTH | |------|------------|--------|------------| | 9997 | 2014-03-18 | 99.17 | 2 | | 9981 | 2014-03-16 | 71.44 | 2 | | 9979 | 2014-03-16 | -94.60 | 3 | | 9977 | 2014-03-16 | -6.96 | 3 | | 9971 | 2014-03-15 | -65.95 | 3 | | 9964 | 2014-03-15 | 15.13 | 2 | | 9962 | 2014-03-15 | 17.47 | 2 | | 9960 | 2014-03-15 | -3.55 | 1 | | 9959 | 2014-03-14 | 32.00 | 1 |

105. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 5. Finding the length of a series | ID | VALUE_DATE | AMOUNT | LENGTH | |------|------------|--------|------------| | 9997 | 2014-03-18 | +99.17 | 2 | | 9981 | 2014-03-16 | +71.44 | 2 | | 9979 | 2014-03-16 | -94.60 | 3 | | 9977 | 2014-03-16 | -6.96 | 3 | | 9971 | 2014-03-15 | -65.95 | 3 | | 9964 | 2014-03-15 | 15.13 | 2 | | 9962 | 2014-03-15 | 17.47 | 2 | | 9960 | 2014-03-15 | -3.55 | 1 | | 9959 | 2014-03-14 | 32.00 | 1 |

107. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 5. Finding the length of a series | ID | VALUE_DATE | AMOUNT | LENGTH | |------|------------|--------|------------| | 9997 | 2014-03-18 | 99.17 | 2 | | 9981 | 2014-03-16 | 71.44 | 2 | | 9979 | 2014-03-16 | -94.60 | 3 | | 9977 | 2014-03-16 | -6.96 | 3 | | 9971 | 2014-03-15 | -65.95 | 3 | | 9964 | 2014-03-15 | +15.13 | 2 | | 9962 | 2014-03-15 | +17.47 | 2 | | 9960 | 2014-03-15 | -3.55 | 1 | | 9959 | 2014-03-14 | 32.00 | 1 |

109. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 5. Finding the length of a series | ID | VALUE_DATE | AMOUNT | LENGTH | |------|------------|--------|------------| | 9997 | 2014-03-18 | 99.17 | 2 | | 9981 | 2014-03-16 | 71.44 | 2 | | 9979 | 2014-03-16 | -94.60 | 3 | | 9977 | 2014-03-16 | -6.96 | 3 | | 9971 | 2014-03-15 | -65.95 | 3 | | 9964 | 2014-03-15 | 15.13 | 2 | | 9962 | 2014-03-15 | 17.47 | 2 | | 9960 | 2014-03-15 | -3.55 | 1 | | 9959 | 2014-03-14 | +32.00 | 1 |

110. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 5. Finding the length of a series | ID | AMOUNT | SIGN | RN | |------|--------|------|----| | 9997 | 99.17 | 1 | 1 | | 9981 | 71.44 | 1 | 2 | | 9979 | -94.60 | -1 | 3 | | 9977 | -6.96 | -1 | 4 | | 9971 | -65.95 | -1 | 5 | | 9964 | 15.13 | 1 | 6 | | 9962 | 17.47 | 1 | 7 | | 9960 | -3.55 | -1 | 8 | | 9959 | 32.00 | 1 | 9 |

111. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 5. Finding the length of a series That’s easy SELECT id, amount, sign(amount) AS sign, row_number() OVER (ORDER BY id DESC) AS rn FROM trx

112. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 5. Finding the length of a series | ID | AMOUNT | SIGN | RN | LO | HI | |------|--------|------|----|----|----| | 9997 | 99.17 | 1 | 1 | 1 | | | 9981 | 71.44 | 1 | 2 | | 2 | | 9979 | -94.60 | -1 | 3 | 3 | | | 9977 | -6.96 | -1 | 4 | | | | 9971 | -65.95 | -1 | 5 | | 5 | | 9964 | 15.13 | 1 | 6 | 6 | | | 9962 | 17.47 | 1 | 7 | | 7 | | 9960 | -3.55 | -1 | 8 | 8 | 8 | | 9959 | 32.00 | 1 | 9 | 9 | 9 |

118. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 5. Finding the length of a series LEAD() and LAG() SELECT lag(v) OVER (ORDER BY v), v, lead(v) OVER (ORDER BY v) FROM ( VALUES (1), (2), (3), (4) ) t(v)

119. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 5. Finding the length of a series SELECT trx.*, CASE WHEN lag(sign) OVER (ORDER BY id DESC) != sign THEN rn END AS lo, CASE WHEN lead(sign) OVER (ORDER BY id DESC) != sign THEN rn END AS hi, FROM trx

120. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 5. Finding the length of a series SELECT -- With NULL handling... trx.*, CASE WHEN coalesce(lag(sign) OVER (ORDER BY id DESC), 0)!=sign THEN rn END AS lo, CASE WHEN coalesce(lead(sign) OVER (ORDER BY id DESC), 0)!=sign THEN rn END AS hi, FROM trx

121. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 5. Finding the length of a series | ID | AMOUNT | SIGN | RN | LO | HI | |------|--------|------|----|----|----| | 9997 | 99.17 | 1 | 1 | 1 | 2 | | 9981 | 71.44 | 1 | 2 | 1 | 2 | | 9979 | -94.60 | -1 | 3 | 3 | 5 | | 9977 | -6.96 | -1 | 4 | 3 | 5 | | 9971 | -65.95 | -1 | 5 | 3 | 5 | | 9964 | 15.13 | 1 | 6 | 6 | 7 | | 9962 | 17.47 | 1 | 7 | 6 | 7 | | 9960 | -3.55 | -1 | 8 | 8 | 8 | | 9959 | 32.00 | 1 | 9 | 9 | 9 |

127. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 5. Finding the length of a series SELECT trx.*, last_value (lo) IGNORE NULLS OVER ( ORDER BY id DESC ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS lo, first_value(hi) IGNORE NULLS OVER ( ORDER BY id DESC ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) AS hi FROM trx

128. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 5. Finding the length of a series SELECT -- With NULL handling... trx.*, coalesce(last_value (lo) IGNORE NULLS OVER ( ORDER BY id DESC ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW), rn) AS lo, coalesce(first_value(hi) IGNORE NULLS OVER ( ORDER BY id DESC ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING), rn) AS hi FROM trx

130. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 5. Finding the length of a series | ID | AMOUNT | SIGN | RN | LO | HI | |------|--------|------|----|----|----| | 9997 | 99.17 | 1 | 1 | 1 | | | 9981 | 71.44 | 1 | 2 | | 2 | | 9979 | -94.60 | -1 | 3 | 3 | 5 | | 9977 | -6.96 | -1 | 4 | 3 | 5 | | 9971 | -65.95 | -1 | 5 | 3 | 5 | | 9964 | 15.13 | 1 | 6 | 6 | | | 9962 | 17.47 | 1 | 7 | | 7 | | 9960 | -3.55 | -1 | 8 | 8 | 8 | | 9959 | 32.00 | 1 | 9 | 9 | 9 |

132. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 5. Finding the length of a series | ID | AMOUNT | SIGN | RN | LO | HI | LENGTH| |------|--------|------|----|----|----|-------| | 9997 | 99.17 | 1 | 1 | 1 | 2 | 2 | | 9981 | 71.44 | 1 | 2 | 1 | 2 | 2 | | 9979 | -94.60 | -1 | 3 | 3 | 5 | 3 | | 9977 | -6.96 | -1 | 4 | 3 | 5 | 3 | | 9971 | -65.95 | -1 | 5 | 3 | 5 | 3 | | 9964 | 15.13 | 1 | 6 | 6 | 7 | 2 | | 9962 | 17.47 | 1 | 7 | 6 | 7 | 2 | | 9960 | -3.55 | -1 | 8 | 8 | 8 | 1 | | 9959 | 32.00 | 1 | 9 | 9 | 9 | 1 |

133. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 5. Finding the length of a series WITH trx1(id, amount, sign, rn) AS ( SELECT id, amount, sign(amount), row_number() OVER (ORDER BY id DESC) FROM trx ), trx2(id, amount, sign, rn, lo, hi) AS ( SELECT trx1.*, CASE WHEN coalesce(lag(sign) OVER (ORDER BY id DESC), 0) != sign THEN rn END, CASE WHEN coalesce(lead(sign) OVER (ORDER BY id DESC), 0) != sign THEN rn END FROM trx1 ) SELECT trx2.*, 1 - last_value (lo) IGNORE NULLS OVER (ORDER BY id DESC ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) + first_value(hi) IGNORE NULLS OVER (ORDER BY id DESC ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) FROM trx2

135. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 6. The subset sum problem with SQL What is the subset sum problem? Explanation: https://xkcd.com/287 (cannot include comic for © reasons. Please, don’t use CC-BY SA NC without an actual commercial offering!) Boring explanation: https://en.wikipedia.org/wiki/Subset_sum_problem

136. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 6. The subset sum problem with SQL For each of these... | ID | TOTAL | |----|-------| | 1 | 25150 | | 2 | 19800 | | 3 | 27511 |

137. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 6. The subset sum problem with SQL ... find the closest sum from these... | ID | ITEM | |------|-------| | 1 | 7120 | | 2 | 8150 | | 3 | 8255 | | 4 | 9051 | | 5 | 1220 | | 6 | 12515 | | 7 | 13555 | | 8 | 5221 | | 9 | 812 | | 10 | 6562 | For each of these... | ID | TOTAL | |----|-------| | 1 | 25150 | | 2 | 19800 | | 3 | 27511 |

138. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 Desired result: 6. The subset sum problem with SQL | TOTAL | BEST | CALCULATION |-------|-------|-------------------------------- | 25150 | 25133 | 7120 + 8150 + 9051 + 812 | 19800 | 19768 | 1220 + 12515 + 5221 + 812 | 27511 | 27488 | 8150 + 8255 + 9051 + 1220 + 812

141. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 6. The subset sum problem with SQL -- All the possible 2N sums WITH sums(sum, max_id, calc) AS (...) -- Find the best sum per “TOTAL” SELECT totals.total, something_something(total - sum) AS best, something_something(total - sum) AS calc FROM draw_the_rest_of_the_*bleep*_owl

142. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 Maybe, if I just hide, the query will go away…? Image credit: https://www.flickr.com/photos/12023825@N04/2898021822 By Peter. License CC-BY SA 2.0

143. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 6. The subset sum problem with SQL What are the possible sums? All the single-item sums | ID | ITEM | |------|-------| | 1 | 7120 | | 2 | 8150 | | 3 | 8255 | | 4 | 9051 | | 5 | 1220 | | 6 | 12515 | | 7 | 13555 | | 8 | 5221 | | 9 | 812 | | 10 | 6562 | “Stack” SUMS(1:10)

144. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 6. The subset sum problem with SQL What are the possible sums? All the single-item sums | ID | ITEM | |------|-------| | 1 | 7120 | | 2 | 8150 | | 3 | 8255 | | 4 | 9051 | | 5 | 1220 | | 6 | 12515 | | 7 | 13555 | | 8 | 5221 | | 9 | 812 | | 10 | 6562 | “Stack” { 7120 } x SUMS(2:10)

145. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 6. The subset sum problem with SQL What are the possible sums? All the single-item sums | ID | ITEM | |------|-------| | 1 | 7120 | | 2 | 8150 | | 3 | 8255 | | 4 | 9051 | | 5 | 1220 | | 6 | 12515 | | 7 | 13555 | | 8 | 5221 | | 9 | 812 | | 10 | 6562 | “Stack” { 7120 + 8150 } x SUMS(3:10)

146. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 6. The subset sum problem with SQL What are the possible sums? All the single-item sums | ID | ITEM | |------|-------| | 1 | 7120 | | 2 | 8150 | | 3 | 8255 | | 4 | 9051 | | 5 | 1220 | | 6 | 12515 | | 7 | 13555 | | 8 | 5221 | | 9 | 812 | | 10 | 6562 | “Stack” { 7120 + 1220 } x SUMS(6:10)

147. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 6. The subset sum problem with SQL What are the possible sums? All the single-item sums | ID | ITEM | IN_THE_SET | |------|-------|------------| | 1 | 7120 | 1 | | 2 | 8150 | 0 | | 3 | 8255 | 0 | | 4 | 9051 | 0 | | 5 | 1220 | 1 | | 6 | 12515 | _ | | 7 | 13555 | _ | | 8 | 5221 | _ | | 9 | 812 | _ | | 10 | 6562 | _ | “Stack” { 7120 + 1220 } x SUMS(6:10)

148. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 -- All the possible 2N sums WITH sums(sum, id, calc) AS ( -- First iteration SELECT item, id, to_char(item) FROM items )

149. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 -- All the possible 2N sums WITH sums(sum, id, calc) AS ( -- First iteration SELECT item, id, to_char(item) FROM items -- Recursion UNION ALL SELECT item + sum, items.id, calc || ' + ' || item FROM sums JOIN items ON sums.id < items.id )

154. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 6. The subset sum problem with SQL ... find the closest sum from these... | ID | ITEM | |------|-------| | 1 | 7120 | | 2 | 8150 | | 3 | 8255 | | 4 | 9051 | | 5 | 1220 | | 6 | 12515 | | 7 | 13555 | | 8 | 5221 | | 9 | 812 | | 10 | 6562 | For each of these... | ID | TOTAL | |------|-------| | 1 | 25150 | | 2 | 19800 | | 3 | 27511 |

155. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 6. The subset sum problem with SQL -- All the possible 2N sums WITH sums(sum, max_id, calc) AS (...) -- Find the best sum per “TOTAL” SELECT totals.id, totals.total, min(abs(total - sum)) AS best_diff FROM totals CROSS JOIN sums GROUP BY totals.id, totals.total

156. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 6. The subset sum problem with SQL -- All the possible 2N sums WITH sums(sum, max_id, calc) AS (...) -- Find the best sum per “TOTAL” SELECT totals.id, totals.total, min(abs(total - sum)) AS best_diff FROM totals CROSS JOIN sums GROUP BY totals.id, totals.total

157. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 R × S Ranks = {A, K, Q, J, 10, 9, 8, 7, 6, 5, 4, 3, 2} Suits = {♠, ♥, ♦, ♣} Ranks × Suits = { (A, ♠), (A, ♥), (A, ♦), (A, ♣), (K, ♠), ..., (2, ♠), (2, ♥), (2, ♦), (2, ♣) } By Trainler - Own work, CC BY 3.0, https://commons.wikimedia.org/w/index.php?curid=7104281 What’s this CROSS JOIN?

158. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 6. The subset sum problem with SQL SELECT totals.id, totals.total, min (sum) KEEP ( DENSE_RANK FIRST ORDER BY abs(total - sum) ) AS best, min (calc) KEEP ( DENSE_RANK FIRST ORDER BY abs(total - sum) ) AS calc, FROM totals CROSS JOIN sums GROUP BY totals.id, totals.total

159. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 WITH sums(sum, id, calc) AS ( SELECT item, id, to_char(item) FROM items UNION ALL SELECT item + sum, items.id, calc || ' + ' || item FROM sums JOIN items ON sums.id < items.id ) SELECT totals.id, totals.total, min (sum) KEEP ( DENSE_RANK FIRST ORDER BY abs(total - sum) ) AS best, min (calc) KEEP ( DENSE_RANK FIRST ORDER BY abs(total - sum) ) AS calc, FROM totals CROSS JOIN sums GROUP BY totals.id, totals.total

160. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 Electrified? Image credit: https://www.flickr.com/photos/battlecreekcvb/4420249105 By Battle Creek CVB. License CC-BY 2.0 (Modified: Faces «anonymised»)

162. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 7. Capping a running total | DATE | AMOUNT | |------------|--------| | 2012-01-01 | 800 | | 2012-02-01 | 1900 | | 2012-03-01 | 1750 | | 2012-04-01 | -20000 | | 2012-05-01 | 900 | | 2012-06-01 | 3900 | | 2012-07-01 | -2600 | | 2012-08-01 | -2600 | | 2012-09-01 | 2100 | | 2012-10-01 | -2400 | | 2012-11-01 | 1100 | | 2012-12-01 | 1300 |

163. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 7. Capping a running total | DATE | AMOUNT | TOTAL | |------------|--------|-------| | 2012-01-01 | 800 | 800 | | 2012-02-01 | 1900 | 2700 | | 2012-03-01 | 1750 | 4450 | | 2012-04-01 | -20000 | 0 | | 2012-05-01 | 900 | 900 | | 2012-06-01 | 3900 | 4800 | | 2012-07-01 | -2600 | 2200 | | 2012-08-01 | -2600 | 0 | | 2012-09-01 | 2100 | 2100 | | 2012-10-01 | -2400 | 0 | | 2012-11-01 | 1100 | 1100 | | 2012-12-01 | 1300 | 2400 |

164. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 7. Capping a running total | DATE | AMOUNT | TOTAL | |------------|--------|-------| | 2012-01-01 | 800 | 800 | GREATEST(0, 800) | 2012-02-01 | 1900 | 2700 | GREATEST(0, 2700) | 2012-03-01 | 1750 | 4450 | GREATEST(0, 4450) | 2012-04-01 | -20000 | 0 | GREATEST(0, -15550) | 2012-05-01 | 900 | 900 | GREATEST(0, 900) | 2012-06-01 | 3900 | 4800 | GREATEST(0, 4800) | 2012-07-01 | -2600 | 2200 | GREATEST(0, 2200) | 2012-08-01 | -2600 | 0 | GREATEST(0, -400) | 2012-09-01 | 2100 | 2100 | GREATEST(0, 2100) | 2012-10-01 | -2400 | 0 | GREATEST(0, -300) | 2012-11-01 | 1100 | 1100 | GREATEST(0, 1100) | 2012-12-01 | 1300 | 2400 | GREATEST(0, 2400)

165. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 7. Capping a running total | DATE | AMOUNT | TOTAL | |------------|--------|-------| | 2012-01-01 | 800 | 800 | GREATEST(0, 800) | 2012-02-01 | 1900 | 2700 | GREATEST(0, 2700) | 2012-03-01 | 1750 | 4450 | GREATEST(0, 4450) | 2012-04-01 | -20000 | 0 | GREATEST(0, -15550) | 2012-05-01 | 900 | 900 | GREATEST(0, 900) | 2012-06-01 | 3900 | 4800 | GREATEST(0, 4800) | 2012-07-01 | -2600 | 2200 | GREATEST(0, 2200) | 2012-08-01 | -2600 | 0 | GREATEST(0, -400) | 2012-09-01 | 2100 | 2100 | GREATEST(0, 2100) | 2012-10-01 | -2400 | 0 | GREATEST(0, -300) | 2012-11-01 | 1100 | 1100 | GREATEST(0, 1100) | 2012-12-01 | 1300 | 2400 | GREATEST(0, 2400)

170. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 7. Capping a running total How to do it? 1. Window functions? Probably not possible 2. Recursive SQL? Not geeky enough 3. Obscure, vendor-specific SQL?

171. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 7. Capping a running total How to do it? 1. Window functions? Probably not possible 2. Recursive SQL? Not geeky enough 3. Obscure, vendor-specific SQL? Jackpot. Someone else will maintain it.

173. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 7. Capping a running total SELECT ... FROM some_table -- Put this after any table MODEL ... Oracle MODEL: Spreadsheet SQL!

174. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 7. Capping a running total MODEL -- The spreadsheet dimensions DIMENSION BY ... -- The spreadsheet cell type MEASURES ... -- The spreadsheet formulas RULES ... Oracle MODEL clause

179. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 7. Capping a running total SELECT * FROM ( SELECT date, amount, 0 AS total FROM amounts ) MODEL DIMENSION BY (row_number() OVER (ORDER BY date) AS rn) -- Individually enumerate each row with a row number

180. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 7. Capping a running total SELECT * FROM ( SELECT date, amount, 0 AS total FROM amounts ) MODEL DIMENSION BY (row_number() OVER (ORDER BY date) AS rn) MEASURES (date, amount, total) -- Each «cell» contains these three values

181. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 7. Capping a running total SELECT * FROM ( SELECT date, amount, 0 AS total FROM amounts ) MODEL DIMENSION BY (row_number() OVER (ORDER BY date) AS rn) MEASURES (date, amount, total) RULES ( total[any] = greatest(0, total[cv(rn) - 1] + amount[cv(rn)]) ) -- «simple» rule based on cv(rn) (cv = current value)

182. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 7. Capping a running total SELECT * FROM ( SELECT date, amount, 0 AS total FROM amounts ) MODEL DIMENSION BY (row_number() OVER (ORDER BY date) AS rn) MEASURES (date, amount, total) RULES ( total[any] = greatest(0, -- Getting NULLs right coalesce(total[cv(rn) - 1], 0) + amount[cv(rn)]) ) -- «simple» rule based on cv(rn) (cv = current value)

184. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 7. Capping a running total Read the whitepaper for more details: http://www.oracle.com/technetwork/mi ddleware/bi-foundation/10gr1-twp-bi- dw-sqlmodel-131067.pdf (Google «Oracle MODEL Whitepaper»)

185. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 8. Time series pattern recognition | ID | VALUE_DATE | AMOUNT | LEN | |------|------------|---------|-----| | 9997 | 2014-03-18 | + 99.17 | 1 | | 9981 | 2014-03-16 | - 71.44 | 4 | | 9979 | 2014-03-16 | - 94.60 | 4 | | 9977 | 2014-03-16 | - 6.96 | 4 | | 9971 | 2014-03-15 | - 65.95 | 4 | | 9964 | 2014-03-15 | + 15.13 | 3 | | 9962 | 2014-03-15 | + 17.47 | 3 | | 9960 | 2014-03-15 | + 3.55 | 3 | | 9959 | 2014-03-14 | - 32.00 | 1 |

186. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 8. Time series pattern recognition | ID | VALUE_DATE | AMOUNT | LEN | TRIGGER |------|------------|---------|-----|-------- | 9997 | 2014-03-18 | + 99.17 | 1 | | 9981 | 2014-03-16 | - 71.44 | 4 | | 9979 | 2014-03-16 | - 94.60 | 4 | x | 9977 | 2014-03-16 | - 6.96 | 4 | | 9971 | 2014-03-15 | - 65.95 | 4 | | 9964 | 2014-03-15 | + 15.13 | 3 | | 9962 | 2014-03-15 | + 17.47 | 3 | | 9960 | 2014-03-15 | + 3.55 | 3 | | 9959 | 2014-03-14 | - 32.00 | 1 |

187. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 | ID | VALUE_DATE | AMOUNT | LEN | TRIGGER |------|------------|---------|-----|-------- | 9997 | 2014-03-18 | + 99.17 | 1 | | 9981 | 2014-03-16 | - 71.44 | 4 | | 9979 | 2014-03-16 | - 94.60 | 4 | x | 9977 | 2014-03-16 | - 6.96 | 4 | | 9971 | 2014-03-15 | - 65.95 | 4 | | 9964 | 2014-03-15 | + 15.13 | 3 | | 9962 | 2014-03-15 | + 17.47 | 3 | | 9960 | 2014-03-15 | + 3.55 | 3 | | 9959 | 2014-03-14 | - 32.00 | 1 | 8. Time series pattern recognition

188. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 8. Time series pattern recognition Trigger on the 3rd repetition of an event if the event occurs more than 3 times.

189. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 8. Time series pattern recognition How to do it? 1. Window functions? Probably not possible 2. Recursive SQL? Not geeky enough 3. Obscure, vendor-specific SQL? Jackpot. Someone else will maintain it.

190. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 8. Time series pattern recognition SELECT ... FROM some_table -- Put this after any table to pattern-match -- the table’s contents MATCH_RECOGNIZE (...) Oracle 12c MATCH_RECOGNIZE!

191. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 8. Time series pattern recognition SELECT * FROM series MATCH_RECOGNIZE ( ORDER BY ... -- Pattern matching is done in this order )

192. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 8. Time series pattern recognition SELECT * FROM series MATCH_RECOGNIZE ( ORDER BY ... MEASURES ... -- These are the columns produced by matches )

193. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 8. Time series pattern recognition SELECT * FROM series MATCH_RECOGNIZE ( ORDER BY ... MEASURES ... ALL ROWS PER MATCH -- A short specification of what rows are -- returned from each match )

194. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 8. Time series pattern recognition SELECT * FROM series MATCH_RECOGNIZE ( ORDER BY ... MEASURES ... ALL ROWS PER MATCH PATTERN (...) -- «Regular expressions» of events to match )

195. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 8. Time series pattern recognition SELECT * FROM series MATCH_RECOGNIZE ( ORDER BY ... MEASURES ... ALL ROWS PER MATCH PATTERN (...) DEFINE ... -- The definitions of «what is an event» )

197. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 8. Time series pattern recognition | ID | VALUE_DATE | AMOUNT | TRIGGER |------|------------|---------|-------- | 9997 | 2014-03-18 | + 99.17 | | 9981 | 2014-03-16 | - 71.44 | | 9979 | 2014-03-16 | - 94.60 | x | 9977 | 2014-03-16 | - 6.96 | | 9971 | 2014-03-15 | - 65.95 | | 9964 | 2014-03-15 | + 15.13 | | 9962 | 2014-03-15 | + 17.47 | | 9960 | 2014-03-15 | + 3.55 | | 9959 | 2014-03-14 | - 32.00 |

198. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 8. Time series pattern recognition SELECT * FROM series MATCH_RECOGNIZE ( ORDER BY id MEASURES ... ALL ROWS PER MATCH PATTERN (...) DEFINE ... )

199. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 8. Time series pattern recognition SELECT * FROM series MATCH_RECOGNIZE ( ORDER BY id MEASURES classifier() AS trg ALL ROWS PER MATCH PATTERN (...) DEFINE ... )

200. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 8. Time series pattern recognition SELECT * FROM series MATCH_RECOGNIZE ( ORDER BY id MEASURES classifier() AS trg ALL ROWS PER MATCH PATTERN (S (R X R+)?) DEFINE ... )

201. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 8. Time series pattern recognition SELECT * FROM series MATCH_RECOGNIZE ( ORDER BY id MEASURES classifier() AS trg ALL ROWS PER MATCH PATTERN (S (R X R+)?) DEFINE R AS sign(R.amount) = prev(sign(R.amount)), X AS sign(X.amount) = prev(sign(X.amount)) )

202. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 8. Time series pattern recognition SELECT * FROM series MATCH_RECOGNIZE ( ORDER BY id MEASURES classifier() AS trg ALL ROWS PER MATCH PATTERN (S (R X R+)?) DEFINE R AS sign(R.amount) = prev(sign(R.amount)), X AS sign(X.amount) = prev(sign(X.amount)) )

203. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 8. Time series pattern recognition | ID | VALUE_DATE | AMOUNT | TRG | |------|------------|---------|-----| | 9997 | 2014-03-18 | + 99.17 | S | | 9981 | 2014-03-16 | - 71.44 | R | | 9979 | 2014-03-16 | - 94.60 | X | | 9977 | 2014-03-16 | - 6.96 | R | | 9971 | 2014-03-15 | - 65.95 | S | | 9964 | 2014-03-15 | + 15.13 | S | | 9962 | 2014-03-15 | + 17.47 | S | | 9960 | 2014-03-15 | + 3.55 | S | | 9959 | 2014-03-14 | - 32.00 | S | PATTERN (S (R X R+)?)

207. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 8. Time series pattern recognition SELECT id, value_date, amount, CASE trg WHEN 'X' THEN 'X' END trg FROM series MATCH_RECOGNIZE ( ORDER BY id MEASURES classifier() AS trg ALL ROWS PER MATCH PATTERN (S (R X R+)?) DEFINE R AS sign(R.amount) = prev(sign(R.amount)), X AS sign(X.amount) = prev(sign(X.amount)) )

208. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 8. Time series pattern recognition | ID | VALUE_DATE | AMOUNT | TRG | |------|------------|---------|-----| | 9997 | 2014-03-18 | + 99.17 | | | 9981 | 2014-03-16 | - 71.44 | | | 9979 | 2014-03-16 | - 94.60 | X | | 9977 | 2014-03-16 | - 6.96 | | | 9971 | 2014-03-15 | - 65.95 | | | 9964 | 2014-03-15 | + 15.13 | | | 9962 | 2014-03-15 | + 17.47 | | | 9960 | 2014-03-15 | + 3.55 | | | 9959 | 2014-03-14 | - 32.00 | |

210. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 8. Time series pattern recognition Read the whitepaper for more details: http://www.oracle.com/ocom/groups/p ublic/@otn/documents/webcontent/19 65433.pdf (Google «Oracle MATCH_RECOGNIZE Whitepaper»)

212. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 9. Pivoting and unpivoting | NAME | TITLE | RATING | |-----------|-----------------|--------| | A. GRANT | ANNIE IDENTITY | G | | A. GRANT | DISCIPLE MOTHER | PG | | A. GRANT | GLORY TRACY | PG-13 | | A. HUDSON | LEGEND JEDI | PG | | A. CRONYN | IRON MOON | PG | | A. CRONYN | LADY STAGE | PG | | B. WALKEN | SIEGE MADRE | R |

213. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 Pivoting 9. Pivoting and unpivoting | NAME | NC-17 | PG | G | PG-13 | R | |-----------|-------|-----|-----|-------|-----| | A. GRANT | 3 | 6 | 5 | 3 | 1 | | A. HUDSON | 12 | 4 | 7 | 9 | 2 | | A. CRONYN | 6 | 9 | 2 | 6 | 4 | | B. WALKEN | 8 | 8 | 4 | 7 | 3 | | B. WILLIS | 5 | 5 | 14 | 3 | 6 | | C. DENCH | 6 | 4 | 5 | 4 | 5 | | C. NEESON | 3 | 8 | 4 | 7 | 3 |

214. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 Unpivoting 9. Pivoting and unpivoting | NAME | RATING | COUNT | |-----------|--------|-------| | A. GRANT | NC-17 | 3 | | A. GRANT | PG | 6 | | A. GRANT | G | 5 | | A. GRANT | PG-13 | 3 | | A. GRANT | R | 6 | | A. HUDSON | NC-17 | 12 | | A. HUDSON | PG | 4 |

216. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 Only PostgreSQL so far 9. Pivoting and unpivoting SELECT first_name, last_name, count(*) FILTER (WHERE rating = 'NC-17') AS "NC-17", count(*) FILTER (WHERE rating = 'PG' ) AS "PG", count(*) FILTER (WHERE rating = 'G' ) AS "G", count(*) FILTER (WHERE rating = 'PG-13') AS "PG-13", count(*) FILTER (WHERE rating = 'R' ) AS "R" FROM actor AS a JOIN film_actor AS fa USING (actor_id) JOIN film AS f USING (film_id) GROUP BY actor_id

217. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 All others 9. Pivoting and unpivoting SELECT first_name, last_name, count(CASE rating WHEN 'NC-17' THEN 1 END) AS "NC-17", count(CASE rating WHEN 'PG' THEN 1 END) AS "PG", count(CASE rating WHEN 'G' THEN 1 END) AS "G", count(CASE rating WHEN 'PG-13' THEN 1 END) AS "PG-13", count(CASE rating WHEN 'R' THEN 1 END) AS "R" FROM actor AS a JOIN film_actor AS fa USING (actor_id) JOIN film AS f USING (film_id) GROUP BY actor_id

218. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 9. Pivoting and unpivoting SELECT actor_id, first_name, last_name, "NC-17", "PG", "G", "PG-13", "R" FROM ( SELECT actor_id, first_name, last_name, rating FROM actor a JOIN film_actor fa USING (actor_id) JOIN film f USING (film_id) ) PIVOT ( count(*) FOR rating IN ( 'NC-17' AS "NC-17", 'PG' AS "PG", 'G' AS "G", 'PG-13' AS "PG-13", 'R' AS "R" ) )

219. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 9. Pivoting and unpivoting SELECT something, something FROM some_table PIVOT ( count(*) FOR rating IN ( 'NC-17' AS "NC-17", 'PG' AS "PG", 'G' AS "G", 'PG-13' AS "PG-13", 'R' AS "R" ) )

220. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 9. Pivoting and unpivoting SELECT something, something FROM some_table UNPIVOT ( count FOR rating IN ( "NC-17" AS 'NC-17', "PG" AS 'PG', "G" AS 'G', "PG-13" AS 'PG-13', "R" AS 'R' ) )

221. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 9. Pivoting and unpivoting Pivoting: Values from a single column become columns containing aggregations Unpivoting: Columns become values in a single column

222. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 9. Pivoting and unpivoting Image credit: https://www.flickr.com/photos/jakerust/16661140289 By GotCredit. License CC-BY 2.0

224. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 10. Abusing XML and JSON Image credit: https://www.flickr.com/photos/bensonkua/6326968245 By Benson Kua. License CC-BY SA 2.0 First, a word of truth

229. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 10. Abusing XML and JSON <actors> <actor> <first-name>Bud</first-name> <last-name>Spencer</last-name> <films>God Forgives... I Don’t, Double Trouble, They Call Him Bulldozer</films> </actor> <actor> <first-name>Terence</first-name> <last-name>Hill</last-name> <films>God Forgives... I Don’t, Double Trouble, Lucky Luke</films> </actor> </actors>

230. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 10. Abusing XML and JSON WITH RECURSIVE x(v) AS (SELECT '...'::xml), actors( actor_id, first_name, last_name, films ) AS (...), films( actor_id, first_name, last_name, film_id, film ) AS (...) SELECT * FROM films

231. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 10. Abusing XML and JSON WITH RECURSIVE x(v) AS (SELECT ' <actors> <actor> <first-name>Bud</first-name> <last-name>Spencer</last-name> <films>God Forgives... I Don’t, Double Trouble, They Call Him Bulldozer</films> </actor> <actor> <first-name>Terence</first-name> <last-name>Hill</last-name> <films>God Forgives... I Don’t, Double Trouble, Lucky Luke</films> </actor> </actors>'::xml), actors(actor_id, first_name, last_name, films) AS (...), films(actor_id, first_name, last_name, film_id, film) AS (...) SELECT * FROM films

232. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 10. Abusing XML and JSON WITH RECURSIVE x(v) AS (SELECT '...'::xml), actors(actor_id, first_name, last_name, films) AS ( SELECT row_number() OVER (), (xpath('//first-name/text()', t.v))[1]::TEXT, (xpath('//last-name/text()' , t.v))[1]::TEXT, (xpath('//films/text()' , t.v))[1]::TEXT FROM unnest(xpath('//actor', (SELECT v FROM x))) t(v) ), films(actor_id, first_name, last_name, film_id, film) AS (...) SELECT * FROM films

233. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 10. Abusing XML and JSON WITH RECURSIVE x(v) AS (SELECT '...'::xml), actors(actor_id, first_name, last_name, films) AS (...), films(actor_id, first_name, last_name, film_id, film) AS ( SELECT actor_id, first_name, last_name, 1, regexp_replace(films, ',.+', '') FROM actors UNION ALL SELECT actor_id, a.first_name, a.last_name, f.film_id + 1, regexp_replace(a.films, '.*' || f.film || ', ?(.*?)(,.+)?', '1') FROM films AS f JOIN actors AS a USING (actor_id) WHERE a.films NOT LIKE '%' || f.film ) SELECT * FROM films

235. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 10 SQL tricks to convince you SQL is awesome 1. Everything is a table 2. Data generation with recursive SQL 3. Running total calculations 4. Finding the length of a series 5. Finding the largest series with no gaps 6. The subset sum problem with SQL 7. Capping a running total 8. Time series pattern recognition 9. Pivoting and unpivoting 10.Abusing XML and JSON (don’t do this at home)

236. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 10 SQL tricks to convince you SQL is awesome Noun awe (uncountable) 1. A feeling of fear and reverence. 2. A feeling of amazement.

237. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 10 SQL tricks to convince you SQL is awesome Noun awe (uncountable) 1. A feeling of fear and reverence. 2. A feeling of amazement.

238. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 10 SQL tricks to convince you SQL is awesome Noun maze (plural mazes) 1. A labyrinth; a puzzle consisting of a complicated network of paths or passages, the aim of which is to find one's way. 2. Something made up of many confused or conflicting elements; a tangle. 3. Confusion of thought; perplexity; uncertainty; state of bewilderment.

239. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 10 SQL tricks to convince you SQL is awesome Noun maze (plural mazes) 1. A labyrinth; a puzzle consisting of a complicated network of paths or passages, the aim of which is to find one's way. 2. Something made up of many confused or conflicting elements; a tangle. 3. Confusion of thought; perplexity; uncertainty; state of bewilderment.

240. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 Why do I talk about SQL? SQL is the only ever successful, mainstream, and general- purpose 4GL (Fourth- Generation Programming Language) And it is awesome!

247. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 So what’s the key takeaway? 1. Can you do it in the database? Yes 2. Can you do it in the database? Yes (... after visiting my 2 day SQL training)

248. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 So what’s the key takeaway? 1. Can you do it in the database? Yes 2. Can you do it in the database? Yes (... after visiting my 2 day SQL training) 3. Can you do it in your database?

249. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 So what’s the key takeaway? 1. Can you do it in the database? Yes 2. Can you do it in the database? Yes (... after visiting my 2 day SQL training) 3. Can you do it in your database? Yes

250. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 So what’s the key takeaway? 1. Can you do it in the database? Yes 2. Can you do it in the database? Yes (... after visiting my 2 day SQL training) 3. Can you do it in your database? Yes (... unless you're using MySQL)

251. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 So what’s the key takeaway? 1. Can you do it in the database? Yes 2. Can you do it in the database? Yes (... after visiting my 2 day SQL training) 3. Can you do it in your database? Yes (... unless you're using MySQL) 4. Should you do it in the database?

252. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 So what’s the key takeaway? 1. Can you do it in the database? Yes 2. Can you do it in the database? Yes (... after visiting my 2 day SQL training) 3. Can you do it in your database? Yes (... unless you're using MySQL) 4. Should you do it in the database? Why not?

253. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 So what’s the key takeaway? 1. Can you do it in the database? Yes 2. Can you do it in the database? Yes (... after visiting my 2 day SQL training) 3. Can you do it in your database? Yes (... unless you're using MySQL) 4. Should you do it in the database? Why not? 5. Do listicles attract attention?

254. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 So what’s the key takeaway? 1. Can you do it in the database? Yes 2. Can you do it in the database? Yes (... after visiting my 2 day SQL training) 3. Can you do it in your database? Yes (... unless you're using MySQL) 4. Should you do it in the database? Why not? 5. Do listicles attract attention? Yes

255. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 So what’s the key takeaway? 1. Can you do it in the database? Yes 2. Can you do it in the database? Yes (... after visiting my 2 day SQL training) 3. Can you do it in your database? Yes (... unless you're using MySQL) 4. Should you do it in the database? Why not? 5. Do listicles attract attention? Yes 6. Will this talk ever end?

256. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 So what’s the key takeaway? 1. Can you do it in the database? Yes 2. Can you do it in the database? Yes (... after visiting my 2 day SQL training) 3. Can you do it in your database? Yes (... unless you're using MySQL) 4. Should you do it in the database? Why not? 5. Do listicles attract attention? Yes 6. Will this talk ever end? Yes

257. Copyright (c) 2009-2016 by Data Geekery GmbH. Slides licensed under CC BY SA 3.0 Thank you Contact me if you want this talk as an in-house training about awesome SQL! Coordinates • Blog: http://blog.jooq.org (excellent Java SQL content) • Twitter: @JavaOOQ / @lukaseder (more lame jokes) • E-Mail: lukas.eder@datageekery.com • Bank account: CH57 8148 7000 0SQL AWSM 7

10 SQL Tricks that You Didn't Think Were Possible

Lukas Eder

10 SQL Tricks that You Didn't Think Were Possible