Fantasy Football - Footballguys Forums
This is a sample guest message. Register a free account today to become a member! Once signed in, you'll be able to participate on this site by adding your own topics and posts, as well as connect with other members through your own private inbox!
Data Guys, Aggregate! (1 Viewer)
- Thread starter Dinsy Ejotuz
- Start date
Gawain
Footballguy
AAABatteries
Footballguy
1. Typically a lot - our usual approach in the past was large models where the users could select whatever dimensions they wanted in a tool like Microstrategy- we have the opportunity to scale it back but not always
2. Mix - what I've said as a strategy for now is to get all the data in Hadoop, stage data marts in Redshift for consumption by Microstrategy (or Tableau) and leave direct access to Hadoop for the analysts/data science teams using Tableau and direct access tools.
AAABatteries
Footballguy
We have some folks evaluating Hawq - my biggest concerns are performance and the limitations we may encounter - based on some of my reading it seems there's some limitations we may encounter.
Curious why you guys went with Spotfire - cost?
AAABatteries
Footballguy
We evaluated it with Tableau - we found Tableau a little easier to use and a little more down the road for Enterprise at the time (3 years ago). I haven't kept up with Qlik's newer versions but Gartner still gives Tableau the lead and our users have been happy so we haven't really questioned the decision. Qlik was a good tool though so you should be happy.
AAABatteries
Footballguy
Have any of you guys looked at DOMO? We did a pretty exhaustive eval on them - they have an interesting product. I loved some features (built-in collaboration) and hated others (ingesting all your data in their solution).
They frame themselves as business management software and not just an analytics tool. Anybody test it out and/or using it?
They frame themselves as business management software and not just an analytics tool. Anybody test it out and/or using it?
Long Ball Larry
Footballguy
Curious what the name of the department is that you guys are in. I am trying to re-organize and re-orient some of our work and I think we should be under a different heading officially.
Osaurus
Footballguy
Keerock
Footballguy
Keerock
Footballguy
We use them all... along with SAS, Pentaho, OBIEE and now implementing an Hadoop Data Lake.
I support them all on the infrastructure side...
Last edited by a moderator:
Keerock
Footballguy
Currently looking to replace QLik with DOMO... not sure where they are in the process.
GregR
Footballguy
I'm not sure the reasoning for it, wasn't involved in this arena at the time. Will see if I can find out though not even sure where the decision was made.
GregR
Footballguy
Heh, feels like I got a completely new one in a lot of ways. So much new stuff to learn, but this is such a great area to get into, I think.
Dinsy Ejotuz
Footballguy
Tableau can do this IIRC. Fairly sure there's a package in R that can do the same, but that's a big learning curve unless you already know how to use R.
Last edited by a moderator:
Insein
Footballguy
Dinsy Ejotuz
Footballguy
Peyton Marino
Footballguy
any tips for a finance guy looking to transition to an analytics role? I majored in math and econ, have a master's in finance, have worked ~4 years in finance. very good in excel and solid in VBA, some light usage of SQL, have taken a couple SAS workshops but there just aren't opportunities in my current role to get practice with it.
online courses? any particular places to look for analytics roles in my area?
online courses? any particular places to look for analytics roles in my area?
Peyton Marino
Footballguy
Gawain
Footballguy
I brought on a guy from JPEG Morgan who took a position for a year and then went to Macy's, but the initial move had to be a paycut. Retail analysis.
Dinsy Ejotuz
Footballguy
We have any ~serious R users in this joint?
I've been working in R most days for the last four months now. To the point where I'm moving out of the "Hey! I know how to get answers!" phase and working to really understand data structures and how to use what's being called the Tidyverse.
More or less readr, tidyr, dplyr, purrr, migrittr, ggplot, shiny, rvest and probably a couple other packages I'm forgetting.
I'd like to be able to stop dropping my R output in Excel and keep it all in R all the way through to the final product -- but that's a stretch right now.
Anyone else use/want to use these?
I've been working in R most days for the last four months now. To the point where I'm moving out of the "Hey! I know how to get answers!" phase and working to really understand data structures and how to use what's being called the Tidyverse.
More or less readr, tidyr, dplyr, purrr, migrittr, ggplot, shiny, rvest and probably a couple other packages I'm forgetting.
I'd like to be able to stop dropping my R output in Excel and keep it all in R all the way through to the final product -- but that's a stretch right now.
Anyone else use/want to use these?
jonessed
Footballguy
I moved to a more strategy role a few years ago, but I was a heavy SAS, Enterprise Miner, SQL, and Tableau user prior to that. I still use Tableau and Base SAS at times. Tableau is pretty handy for on-the-fly data visualization when building a presentation deck.
I see a lot more R users now as well.
I see a lot more R users now as well.
Peyton Marino
Footballguy
Limp Ditka
Footballguy
GregR
Footballguy
I've got a Hive view, 6 columns something like:
ID ... DATE ... Value A ... Value B ... Value C ... Value D
Where the combo of ID and Date are unique.
For each ID I would like to retrieve for each of A/B/C/D the most recent Date that each had a value (i.e. was not NULL), and what that value was. So I'd end up with 9 columns... ID, Date A, Value A, Date B, Value B, etc.
I could do each of A/B/C/D in separate subqueries and join them back together. But is there a more performant way to do it, that can return what I want in a single pass through the data instead of accessing the original view 4 times?
ID ... DATE ... Value A ... Value B ... Value C ... Value D
Where the combo of ID and Date are unique.
For each ID I would like to retrieve for each of A/B/C/D the most recent Date that each had a value (i.e. was not NULL), and what that value was. So I'd end up with 9 columns... ID, Date A, Value A, Date B, Value B, etc.
I could do each of A/B/C/D in separate subqueries and join them back together. But is there a more performant way to do it, that can return what I want in a single pass through the data instead of accessing the original view 4 times?
skol asylum
Footballguy
Finance guy that does the TPS reports. Sounds like we are going to have the opportunity to use Tableau in our area eventually. Most of our reports are from the GL and created in Excel using an OLAP cube and SSAS. Would also like to hone my tech skills. Any recommendations appreciated.
Emerson Biggens
Footballguy
What training/ classes did you do?still in the stages of finalizing, but it's looking like I made this happen
GregR
Footballguy
It is going to be end up being a view, yes. I'm trying to write the query for the view.
Getting the query to work isn't the problem. I'm trying to see if there is a more performant way to do it than I would have in Oracle with a bunch of GROUP BY's to get the max dates. Actually, I know that RANK() is more performant in Hive than GROUP BY for this. But what I'm asking is, I was hoping there was a performance improving way that instead of reading my big data set 4 times, once to get the max date for each of A/B/C/D, that it could process all four on one pass through the data.
Or maybe it will already do that behind the scenes I don't know. I haven't looked at an execution plan yet, was just kind of mapping out the query in my head so far.
Last edited by a moderator:
Insein
Footballguy
Been on Severance for a couple months now. I've been casually looking for jobs but not aggressively. I have another 4 months but it's time to go at it full bore.
I have Tableau skills but it's been almost 2 years since I've used them daily. Anywhere I can brush up on them a bit to jog my memory?
Also any new tools out there that people are looking for their data guys to have?
I have Tableau skills but it's been almost 2 years since I've used them daily. Anywhere I can brush up on them a bit to jog my memory?
Also any new tools out there that people are looking for their data guys to have?
Last edited by a moderator:
TakiToki
Footballguy
Could you unpivot to ID, Value Type and Date; rank by descending Date grouped by ID and Value Type; pivot back out the the top ranked rows
Last edited by a moderator:
Peyton Marino
Footballguy
I took the intro and intermediate courses for R on edx.org. I also took a Python course that MIT offered on the same website. My knowledge of these programs is still on the beginner side.
I would say the biggest factors in my securing this position was a) interviewing skills b) I have a friend who works closely with the hiring manager who put in a strong recommendation for me c) I was able to sell the hiring manager on my general intelligence, ability to learn quickly, and passion for D&A
Insein
Footballguy
That's usually 95% of the hire. Then you just hope whatever they're asking is within your realm of learning quickly. At least that's been my experience. It usually is as the hiring people like to make the job description overly complicated.
Keerock
Footballguy
Have you tried using a materialized view or populating a GTT to pre-collect your data and then query that object at runtime for improved performance?
GregR
Footballguy
Insein
Footballguy
dgreen
Footballguy
Any machine learning experts here? I have a project to take text descriptions of products/goods and classify them into a hierarchical coding structure. At the last minute, I was able to get an intern this Summer with the skills to create a model. He did a great job, however the results aren't good enough yet and the biggest reason is likely that we didn't do the proper work to give him a good enough training set. We simply didn't have the time to do that before the internship started, but I wasn't going to turn down free help. At the most detailed level of the product structure, he's only getting a 48% match rate with the test data. At the highest level, he's up to 65%.
The data is survey response data, so we know it can be messy. We took some quick, easy steps to eliminate data we were confident was bad and then made some assumptions about what data is good to create the training set. We learned a lot of bad data remained in the training set.
My goal right now is to get a sense of what's involved with creating an adequate training set so I can get a cost estimate and try to fund that task. Then we'll look to get someone else to pick up the interns program/model with a better training set and hopefully see better results. My first thought is to take our current training set (230,000 rows) and have people go through, row by row, to either verify or correct the text/code combinations. Also need to figure out how to deal with the misspellings, abbreviations, and other nonsense that comes along with survey response data. We tried a spell correction program, but it didn't improve the model much. And, I'm thinking maybe we shouldn't correct spelling because misspellings are a reality of the data and common misspellings are, I assume, just as good of predictors as properly spelled words (it's not like the model knows or cares about proper spelling; it only cares about correlation).
If anyone has knowledge in this area, I'd love to hear your thoughts and experiences. Or, if anyone knows of some good online training about creating training sets, that would be great. I can expand more, if needed, about the data; just ask questions so I know what's relevant to share rather than rambling on about irrelevant stuff.
The data is survey response data, so we know it can be messy. We took some quick, easy steps to eliminate data we were confident was bad and then made some assumptions about what data is good to create the training set. We learned a lot of bad data remained in the training set.
My goal right now is to get a sense of what's involved with creating an adequate training set so I can get a cost estimate and try to fund that task. Then we'll look to get someone else to pick up the interns program/model with a better training set and hopefully see better results. My first thought is to take our current training set (230,000 rows) and have people go through, row by row, to either verify or correct the text/code combinations. Also need to figure out how to deal with the misspellings, abbreviations, and other nonsense that comes along with survey response data. We tried a spell correction program, but it didn't improve the model much. And, I'm thinking maybe we shouldn't correct spelling because misspellings are a reality of the data and common misspellings are, I assume, just as good of predictors as properly spelled words (it's not like the model knows or cares about proper spelling; it only cares about correlation).
If anyone has knowledge in this area, I'd love to hear your thoughts and experiences. Or, if anyone knows of some good online training about creating training sets, that would be great. I can expand more, if needed, about the data; just ask questions so I know what's relevant to share rather than rambling on about irrelevant stuff.
Long Ball Larry
Footballguy
not sure this is the right place for this, but what the hell.
i am looking for a program (or code suggestion) that can re-calculate financial histories. there could be hundreds of transactions with charges, penalties, interest and payments. the penalties and interest were assessed at specific rates and the penalty and interest amounts assessed depended on the running balance of the various components at the time. problem is that the rates need to be changed retroactively, which is really annoying to deal with from because all the payments have to be re-allocated in accordance with the adjusted rate. so i am trying to figure out a more automated way than individually calculating all of the transactions.
any thoughts? does quickbooks or something similar have a function where you can do that (especially if you can import).
i am looking for a program (or code suggestion) that can re-calculate financial histories. there could be hundreds of transactions with charges, penalties, interest and payments. the penalties and interest were assessed at specific rates and the penalty and interest amounts assessed depended on the running balance of the various components at the time. problem is that the rates need to be changed retroactively, which is really annoying to deal with from because all the payments have to be re-allocated in accordance with the adjusted rate. so i am trying to figure out a more automated way than individually calculating all of the transactions.
any thoughts? does quickbooks or something similar have a function where you can do that (especially if you can import).
Dinsy Ejotuz
Footballguy
Bump.
Dinsy Ejotuz
Footballguy
Not 100% sure what you're looking for, but this sounds like something that would need to be coded.
GregR
Footballguy
So I've been working with what amounts to our data lake infrastructure and ingestion group. It's mainly a keep it running and create data pipelines group. I'm more of a data guy though and as one of the only people on the team with much domain knowledge I'd kind of carved out a niche working projects where I was a technical interface with the business users doing transformations and data integration though it's not our focus. I am going to try to make a change back to a more data centric role though as I still often feel like a data fish in a pond of CS majors. We are really growing our data analytics positions so I'm tooling up for that. Taking some Python courses at present via Coursera that I can get for the first couple for free via work.
Anyone built up their statistics skills for data science, and have any recommendations there?
Anyone built up their statistics skills for data science, and have any recommendations there?
What exactly is a data lake? I keep hearing it mentioned in some large, very slow project at my company but nobody seems to know what they are actually building.
