Skip to content

Select expression #112

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jeremyrsmith wants to merge 10 commits into from
Closed

Select expression #112

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
815b7a2
Change TypeDataset#apply syntax to use a function
jeremyrsmith Feb 18, 2017
82f1ba7
Update tuts for new syntax
jeremyrsmith Feb 18, 2017
fa0ce2f
Coverage
jeremyrsmith Feb 18, 2017
add79ec
More macros!
jeremyrsmith Feb 20, 2017
b58503a
Better error messaging & test coverage
jeremyrsmith Feb 20, 2017
c7550c5
Added Array TypedEncoder
jeremyrsmith Feb 20, 2017
1e6e019
Added functions to selectExpr
jeremyrsmith Feb 22, 2017
d5b8495
Prioritize injection over derivation
jeremyrsmith Feb 22, 2017
df32af4
Added more aggregate functions and misc functions
jeremyrsmith Feb 22, 2017
aa1e55b
Revert prioritization
jeremyrsmith Feb 22, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ val scalatest = "3.0.1"
val shapeless = "2.3.2"
val scalacheck = "1.13.4"

// spark has scalatest and scalactic as a runtime dependency
// which can mess things up if you use a different version in your project
val exclusions = Seq(ExclusionRule("org.scalatest"), ExclusionRule("org.scalactic"))
Copy link
Contributor

@imarios imarios Feb 20, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this fixs the IntelliJ issue with ScalaTest?

Copy link
Contributor Author

@jeremyrsmith jeremyrsmith Feb 20, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep!


lazy val root = Project("frameless", file("." + "frameless")).in(file("."))
.aggregate(core, cats, dataset, docs)
.settings(framelessSettings: _*)
Expand All @@ -22,7 +26,7 @@ lazy val cats = project
.settings(publishSettings: _*)
.settings(libraryDependencies ++= Seq(
"org.typelevel" %% "cats" % catsv,
"org.apache.spark" %% "spark-core" % sparkVersion % "provided"))
"org.apache.spark" %% "spark-core" % sparkVersion % "provided" excludeAll(exclusions: _*)))

lazy val dataset = project
.settings(name := "frameless-dataset")
Expand All @@ -31,8 +35,8 @@ lazy val dataset = project
.settings(framelessTypedDatasetREPL: _*)
.settings(publishSettings: _*)
.settings(libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % sparkVersion % "provided",
"org.apache.spark" %% "spark-sql" % sparkVersion % "provided"
"org.apache.spark" %% "spark-core" % sparkVersion % "provided" excludeAll(exclusions: _*),
"org.apache.spark" %% "spark-sql" % sparkVersion % "provided" excludeAll(exclusions: _*)
))
.dependsOn(core % "test->test;compile->compile")

Expand Down
3 changes: 2 additions & 1 deletion dataset/src/main/scala/frameless/TypedColumn.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -271,6 +271,7 @@ object TypedColumn {
lgen: LabelledGeneric.Aux[T, H],
selector: Selector.Aux[H, K, V]
): Exists[T, K, V] = new Exists[T, K, V] {}

}

implicit class OrderedTypedColumnSyntax[T, U: CatalystOrdered](col: TypedColumn[T, U]) {
Expand All @@ -279,4 +280,4 @@ object TypedColumn {
def >(other: TypedColumn[T, U]): TypedColumn[T, Boolean] = (col.untyped > other.untyped).typed
def >=(other: TypedColumn[T, U]): TypedColumn[T, Boolean] = (col.untyped >= other.untyped).typed
}
}
}
9 changes: 5 additions & 4 deletions dataset/src/main/scala/frameless/TypedDataset.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,11 +62,9 @@ class TypedDataset[T] protected[frameless](val dataset: Dataset[T])(implicit val
*
* It is statically checked that column with such name exists and has type `A`.
*/
def apply[A](column: Witness.Lt[Symbol])(
implicit
exists: TypedColumn.Exists[T, column.T, A],
def apply[A](selector: T => A)(implicit
encoder: TypedEncoder[A]
): TypedColumn[T, A] = col(column)
): TypedColumn[T, A] = macro frameless.macros.ColumnMacros.fromFunction[T, A]

/** Returns `TypedColumn` of type `A` given it's name.
*
Expand Down Expand Up @@ -319,6 +317,9 @@ class TypedDataset[T] protected[frameless](val dataset: Dataset[T])(implicit val
}
}

def selectExpr[B](expr: T => B)(implicit encoder: TypedEncoder[B]): TypedDataset[B] =
macro frameless.macros.ColumnMacros.fromExpr[T, B]

/** Type-safe projection from type T to Tuple2[A,B]
* {{{
* d.select( d('a), d('a)+d('b), ... )
Expand Down
Loading