Spark Streaming
Here is an example of how to do a very simple streaming from and to file with pyspark. https://gist.github.com/sallos-cyber/a14e03da49cc0c873a651628dba4d096
Spark
Frontend
How to serve local web files over HTTP for testing
Recently I got interested in creating my own webapp. If you write a simple html file you can open it with a browser. Notice that in the browser in the
Spark
SPARK configuration
spark.dynamicAllocation.enabled spark.dynamicAllocation.initialExecutors spark.dynamicAllocation.minExecutors spark.dynamicAllocation.maxExecutors spark.shuffle.partitions spark.default.parallelism = spark.executor.instances * spark.executor.cores * 2 maxPartitionBytes Input bytes = 40 GB? Wähle so viele Partitions, so dass die Größe einer Partition <= 200
Understanding the Probability Density Function of the Normal Distribution
A random variable $Z$ is said to have the standard normal distribution, if its probability density function (pdf) is as follows: \[\begin{equation}f_Z(z)=\frac{1}{\sqrt{2\pi}} * \exp(\frac{-z^2}{2}), \quad -\infty<z<\infty\end{equation} \tag{1}\label{eq:eq1} \] This formula
C / C++, Programming
C Pointer and Reference
https://gist.github.com/sallos-cyber/d2bc782f09688beefb4e66068af03174
Big Data Tools, Spark
Pyspark streaming from and to csv-file in Zeppelin: basic code example
Assumptions: zeppelin 10.0, and Spark 3.1.1. I assume Spark runs in one thread on a single machine (local) and Zeppelin runs on the same machine. The Spark-Home variable has been